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FEEDBACK  IN  VIDEOGAME-BASED  ADAPTIVE  TRAINING 


EXECUTIVE  SUMMARY 


Research  Requirement: 

Military,  industry,  and  education  continue  to  investigate  the  use  of  an  increasingly  popular 
instructional  tool  known  as  videogame-based  adaptive  training  (Orvis,  Horn,  &  Belanich,  2008). 
Videogame-based  adaptive  training  has  provided  flexibility  and  adaptability  for  training  in  cost- 
effective  ways.  While  this  method  of  training  may  have  many  benefits  for  the  trainee,  current 
research  has  not  kept  up  to  pace  with  its  implementation. 

One  area  of  research  that  has  not  received  much  attention  is  the  effect  of  feedback  in  videogame- 
based  adaptive  training.  Research  suggests  that  training  alone  is  not  sufficient  to  enhance 
performance  but  that  feedback  is  a  necessary  component  (Komaki,  Heinzmann,  &  Lawson, 

1980)  and  is  critical  for  evaluating  new  skills  (McKendree,  1990).  Additionally,  feedback  has 
been  shown  to  motivate  and  direct  employees’  attention  and  thus  increase  learning  and 
performance  (Kluger  &  DeNisi,  1996).  As  critical  as  feedback  appears  to  be,  few  studies  have 
evaluated  the  real  impact  of  different  types  of  feedback  on  training  perfonnance.  Most  computer 
based  training  systems  provide  some  sort  of  immediate  feedback  but  feedback  content  is  rarely 
explicitly  discussed  (McKendree,  1990).  Another  issue  is  that  the  literature  is  mixed  on  which 
type  of  feedback  is  most  effective  for  learning. 

To  address  this  gap,  competing  theories  in  feedback  frequency  and  sign  were  evaluated  to  determine 
the  most  effective  training  design  features.  The  feedback  intervention  theory  (Kluger  &  DeNisi, 

1996)  supports  the  notion  that  for  a  complex  skill  it  is  optimal  to  provide  trainees  with  infrequent 
feedback.  Contrary  to  this  theory,  Anderson’s  (1983)  ACT-R  theory  suggests  that  frequent  feedback 
is  ideal  for  learning.  Bandura’s  (1986)  self-efficacy  theory  claims  that  people  will  try  harder  or  raise 
their  goals  after  success  (i.e.,  positive  feedback).  However,  control  theory  (Carver  &  Scheier,  1981) 
claims  that  failure  (i.e.,  negative  feedback)  motivates  performance  more  than  success  does. 
Furthermore,  an  individual’s  level  of  feedback  orientation,  which  is  an  individual’s  overall 
acceptance  of  feedback,  will  be  examined  as  a  potential  moderator.  This  research  is  the  first  to 
examine  feedback  orientation  in  a  training  context. 

Procedure: 

The  training  program  used  was  Virtual  Environment  Cultural  Training  for  Operational  Readiness 
(VECTOR),  developed  for  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
by  Chi  Systems,  Inc.  VECTOR  is  designed  to  provide  videogame-based  adaptive  training  in 
interpersonal  skills  through  the  application  of  highly  experiential  and  scenario-based  training. 
Participants  were  asked  to  complete  two  missions  in  VECTOR. 

The  design  of  the  experiment  was  based  on  the  voice-over  feedback  that  was  provided  to  the  trainee. 
The  design  was  a  2  x  2  mixed-design  with  the  dependent  variable  as  a  performance  change  score. 
Participants  were  randomly  assigned  to  one  of  four  conditions:  (1)  frequent  positive  feedback,  (2) 
infrequent  positive  feedback,  (3)  frequent  negative  feedback,  or  (4)  infrequent  negative  feedback.  All 
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feedback  provided  was  based  on  the  trainee’s  actual  performance.  In  addition  to  playing  the 
videogame,  participants  were  asked  to  fill  out  several  paper-and-pencil  measures.  These  measures 
were:  a  multiple-choice  pre-  and  post-test  on  interpersonal  skills,  Feedback  Orientation  Scale, 
manipulation  check  scale,  presence  scale,  attention  scale  and  self-efficacy  scale. 

Findings: 

The  results  reinforce  Anderson’s  ACT-R  model.  That  is,  in  videogame-based  training  scenarios, 
frequent  feedback  leads  to  higher  post  test  scores  than  infrequent  feedback.  It  appears  that 
infrequent  feedback  did  not  provide  enough  cues  for  the  participant  to  detect  and  reject  erroneous 
hypotheses  and  this  in  turn  caused  the  participant  to  compile  incorrect  information  which 
decreased  perfonnance  (Anderson,  1983,  1996,  2000).  Furthermore,  the  results  indicate  no 
support  for  the  main  effect  of  feedback  sign  on  performance,  and  thus  neither  theory  regarding 
the  role  of  feedback  sign  on  training  performance  was  supported. 

The  interaction  between  feedback  frequency  and  feedback  sign  was  also  examined.  The 
interaction  was  significant  and  further  post-hoc  analyses  indicated  that  frequent  feedback  was 
most  beneficial  when  the  feedback  was  negative  rather  than  positive.  Also,  positive  feedback 
leads  to  better  performance  when  it  was  infrequent  rather  than  frequent. 

Lastly,  it  was  predicted  that  feedback  orientation  would  moderate  the  relationship  between 
feedback  intervention  and  performance.  This  prediction  was  not  supported.  In  a  training 
environment,  feedback  orientation  may  be  less  influential  because  trainees  are  expecting  to 
receive  feedback  on  their  perfonnance. 

Utilization  and  Dissemination  of  Findings: 

Instructional  designers  can  use  this  information  to  determine  what  form  of  feedback  is  most 
beneficial  to  training  a  complex  task.  Instructional  designers  need  to  keep  in  mind  that  during  a 
complex  task  the  trainee  requires  frequent  feedback  about  their  perfonnance.  Preferably,  the 
feedback  provided  will  be  constructive  rather  than  just  positive  praise.  Positive  praise  only 
informs  whether  the  goal  was  accomplished.  However,  constructive  feedback  provides 
information  on  how  to  improve  performance.  Lastly,  feedback  should  adapt  to  the  trainee’s 
learning  and  performance.  The  use  of  technology  such  as  computer-based  training,  videogames 
and  web-enabled  training  is  still  an  emerging  field  where  there  is  a  lack  of  theoretical 
background  to  help  develop  training.  This  effort  takes  a  step  toward  resolving  this  problem  to 
allow  practitioners  a  framework  upon  which  to  base  their  training  design. 
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Feedback  in  Videogame -based  Adaptive  Training 

The  field  of  training  has  been  quickly  evolving  due  to  the  implementation  of  new  and  advanced 
technology.  Military,  industry,  and  education  continue  to  investigate  the  use  of  an  increasingly 
popular  instructional  tool  known  as  videogame-based  adaptive  training  (Orvis,  Horn,  & 

Belanich,  2008).  Videogame-based  adaptive  training  involves  a  videogame  platform  where 
automatic  adjustments  are  made  to  problems  or  tasks  based  on  a  trainee’s  performance.  The  use 
of  videogames  has  become  a  popular  medium  for  training  in  the  military  because  they  are 
effective  tools  for  learning  and  understanding  complex  subject  matter  (Ricci,  Salas,  &  Cannon- 
Bowers,  1996).  This  approach  is  also  immersive  and  engaging  in  ways  that  traditional 
workbooks  or  manuals  are  not  and  has  the  capability  to  provide  realistic  feedback  and  multi- 
sensory  information  (Garris,  Ahlers  &  Driskell,  2002;  Tarr,  Morris,  &  Singer,  2002).  Research 
has  also  shown  that  skills  learned  in  game-based  training  environments  transfer  to  real-life 
situations  (Gopher,  Weil,  &  Bareket,  1994;  Knerr,  Simutis,  &  Johnson,  1979).  Additionally, 
videogame-based  training  offers  a  training  medium  that  is  quickly  deployable,  has  low  costs  and  can 
be  easily  administered  without  the  expertise  of  a  trainer.  The  rapid  growth  and  implementation  of 
videogame-based  training  reinforces  the  need  for  a  theoretical  framework  that  guides  the  design  of 
these  tools.  For  this  reason,  this  research  will  examine  an  important  design  element  in  training: 
feedback.  More  specifically,  this  research  tests  conflicting  feedback  theories  regarding  feedback  sign 
and  feedback  frequency.  The  goal  was  to  determine  which  type  of  feedback  (positive,  negative, 
frequent,  or  infrequent)  is  most  effective  for  improving  interpersonal  skills  in  a  videogame-based 
adaptive  training  environment. 

According  to  the  U.S.  Army  Concept  for  the  Human  Dimension  in  Full  Spectrum  Operations 
2015-2024  (TRADOC  Pamphlet  525-3-7,  2008),  there  is  a  need  to  expand  the  skills  of  Soldiers 
beyond  specialty  skills  and  common  warrior  skills  to  include  interpersonal  skills.  More 
specifically,  there  is  a  shift  from  training  for  operations  in  sharply  defined  institutional  chains  of 
command  to  decentralized,  collaborative  operations  requiring  the  use  of  negotiation  skills.  Army 
leaders  need  experience  negotiating  in  the  context  of  ambiguous  authority,  political  turmoil  and 
cultural  differences.  The  future  of  interpersonal  skills  training  requires  the  leader  to  have  access 
to  experiential  learning  opportunities  such  as  scenario-based  vignettes  with  coaches  that  allow 
them  to  practice  adaptable  decision  making  skills.  Research  must  address  these  needs  in  order  to 
provide  Soldiers  with  the  most  effective  training  tools  for  the  future. 

Currently,  most  of  the  research  in  computer-based  training  has  been  focused  on  motor  or  “hard 
skills”  due  to  the  feasibility  in  defining,  measuring,  and  testing  a  well-defined  task  or  domain 
(Schmidt  &  Wulf,  1997;  Swinnen,  Schmidt,  Nicholson  &  Shapiro,  1990;  Todorov  &  Shadmehr, 
1997).  However,  computer-based  training  has  the  potential  to  also  train  “soft  skills”  such  as 
interpersonal  skills.  Interpersonal  skills  refer  to  a  person’s  ability  to  present  oneself  in  a  manner 
that  promotes  positive  relationships  through  social  communications  and  interactions.  This 
includes  the  ability  to  manage  people,  negotiate,  and  give  and  receive  constructive  criticisms. 
Some  organizations  have  begun  to  apply  computer-based  adaptive  training  to  interpersonal  skills 
(Bailey,  1990;  Drew  &  Davidson,  1993;  Ross,  Pollman,  Perry,  Welty  &  Jones,  2001;  Tomey- 
Purta,  1998).  However,  to  date,  the  effectiveness  of  this  training  has  not  been  established. 
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Another  area  of  research  that  requires  further  investigation  is  the  role  of  feedback  in  videogame- 
based  adaptive  training.  Research  suggests  that  training  alone  is  not  sufficient  to  enhance 
performance  but  that  feedback  is  a  necessary  component  (Komaki,  Heinzmann,  &  Lawson, 

1980)  and  is  critical  for  evaluating  new  skills  (McKendree,  1990).  Additionally,  feedback  has 
been  shown  to  motivate  and  direct  employees’  attention  and  thus  increase  learning  and 
performance  (Kluger  &  DeNisi,  1996).  As  critical  as  feedback  appears  to  be,  few  studies  have 
evaluated  the  real  impact  of  different  types  of  feedback  on  training  perfonnance.  Most 
computer-based  training  systems  provide  some  sort  of  immediate  feedback  but  feedback  content 
is  rarely  explicitly  discussed  (McKendree,  1990).  Another  issue  is  that  the  literature  is  mixed  on 
which  type  of  feedback  is  most  effective  for  learning.  There  has  been  some  limited  research  on 
topics  such  as  face-to-face  feedback  versus  electronic  feedback  (Earley,  1988;  Kluger  &  Adler, 
1993;  Kluger  &  DeNisi,  1996).  Beyond  this  there  has  been  little  empirical  work  examining  the 
optimal  use  of  feedback  in  videogame-based  adaptive  training. 

To  address  this  gap  in  the  literature,  the  current  research  compared  trainee  perfonnance  on  a 
videogame-based  adaptive  training  program  for  interpersonal  skills  after  receiving  different  types 
of  feedback.  The  training  program  used  for  this  research  was  the  Virtual  Environment 
Composable  Training  for  Operational  Readiness  (VECTOR)  developed  by  Chi  Systems,  Inc.,  for 
the  U.S.  Anny  Research  Institute  (ARI),  partly  through  an  Army  Small  Business  Innovation 
Research  contract  and  partly  with  ARI  funding.  VECTOR  is  designed  to  provide  game-based 
adaptive  training  in  interpersonal  skills  through  the  application  of  highly  experiential,  scenario- 
based  training  in  a  virtual  environment.  This  type  of  platform  is  in  line  with  what  is 
recommended  by  the  U.S.  Army  Concept  for  the  Human  Dimension  in  Full  Spectrum  (TRADOC 
Pamphlet  525-3-7,  2008),  which  states  the  need  to  expose  Soldiers  to  risk  in  realistic  simulation- 
supported  training  in  order  to  improve  decision  making  skills  without  fear  of  injury. 

Additionally,  competing  theories  in  feedback  frequency  (feedback  intervention  theory  and  ACT- 
R  theory)  and  feedback  sign  (self-efficacy  theory  and  control  theory)  were  evaluated  to 
determine  the  most  effective  training  design  features.  Furthermore,  an  individual’s  level  of 
feedback  orientation,  which  is  an  individual’s  overall  acceptance  of  feedback,  was  examined  as  a 
potential  moderator  between  the  feedback  intervention  and  performance.  It  was  hypothesized 
that  if  trainees  in  the  frequent  feedback  condition  perfonned  better  (i.e.,  improved  posttest  scores 
and  improved  scores  from  Mission  1  to  Mission  2  in  the  videogame)  in  comparison  to  trainees  in 
the  infrequent  feedback  condition  then  the  ACT-R  theory  would  be  supported.  If  trainees  in  the 
positive  feedback  condition  performed  better  than  trainees  in  the  negative  feedback  condition 
then  the  self-efficacy  theory  would  be  supported. 

The  feedback  intervention  theory  (Kluger  &  DeNisi,  1996)  supports  the  notion  that  for  an 
interpersonal  or  complex  skill  it  is  optimal  to  provide  trainees  with  infrequent  feedback. 

Contrary  to  this  theory,  Anderson’s  (1983)  ACT-R  theory  suggests  that  feedback  should  take 
place  immediately  after  every  action,  in  other  words,  frequent  feedback  is  viewed  as  ideal  for 
learning.  By  empirically  examining  these  two  competing  theories  of  feedback,  this  effort  will 
help  instructional  designers  incorporate  the  appropriate  amount  of  feedback  in  their  videogame- 
based  adaptive  training  programs.  This  research  will  also  test  two  competing  theories  regarding 
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feedback  sign.  Bandura’s  (1986)  self-efficacy  theory  claims  that  people  will  try  harder  or  raise 
their  goals  after  success.  This  is  supported  by  research  that  shows  positive  feedback  is  best  for 
improving  performance  on  a  task  (Becker  &  Klimoski,  1989;  Ilgen,  Fisher  &  Taylor,  1979; 
Kluger  &  DeNisi,  2005;  Shrauger  &  Rosenberg,  1970;  Van-Dijk  &  Kluger,  2004).  On  the  other 
hand,  the  control  theory  perspective  (Carver  &  Scheier,  1981)  claims  that  failure  motivates 
performance  and  persistence  more  than  success  does.  Consistent  with  this  approach,  research 
demonstrates  that  negative  feedback  is  beneficial  for  improving  performance  (Atawater,  1995; 
Johnson  &  Ferstl,  1999;  Reilly,  Smither,  &  Vasilopoulos,  1996;  Smither  et  al.,  1995;  Walker  & 
Smither,  1999).  The  current  investigation  will  examine  which  type  of  feedback  sign  is  most 
beneficial  for  learning  an  interpersonal  skill  in  a  videogame -based  adaptive  training 
environment. 

Lastly,  a  key  challenge  in  the  feedback  literature  is  the  need  for  a  better  understanding  of  an 
individual’s  response  to  performance  feedback  (Fedor,  1991)  in  a  videogame-based  adaptive 
training  environment.  Little  research  has  examined  how  responses  to  feedback  may  differ  across 
individuals  (Renn  &  Prien,  1995).  This  is  especially  the  case  concerning  a  person’s  feedback 
orientation,  which  is  an  individual’s  overall  acceptance  of  and  value  placed  on  feedback.  This  is 
an  important  characteristic  to  examine  because  researchers  suggest  that  a  positive  feedback 
orientation  may  lead  to  performance  improvements  (London  &  Smither,  2002;  Smither,  London, 
&  Reilly,  2005).  Prior  to  designing  a  training  program,  knowing  a  trainee’s  feedback  orientation 
may  help  in  determining  what  form  of  feedback  would  lead  to  superior  performance.  This  effort 
is  the  first  to  examine  feedback  orientation  in  a  training  context  and  its  impact  on  feedback 
acceptance  and  perfonnance. 

The  goal  of  this  research  is  to  address  the  following  issues:  Which  is  most  useful  for  learning 
interpersonal  skills  in  a  videogame-based  adaptive  training  environment,  frequent  or  infrequent 
feedback,  positive  or  negative  feedback?  Which  theory  best  explains  this  relationship?  Does 
feedback  orientation  moderate  the  relationship  between  feedback  intervention  and  performance? 

The  next  section  will  review  the  current  training  and  feedback  literature.  The  following  section 
will  consist  of  an  explanation  of  the  methods  used  and  results.  Finally,  the  discussion  section 
will  review  the  theoretical  and  practical  implications  of  this  investigation. 


Training  Interpersonal  Skills 
Current  Trends  in  Interpersonal  Skills  Training 

The  importance  of  training  Soldiers  on  interpersonal  skills  is  emphasized  in  the  U.S.  Army  Concept 
for  the  Human  Dimension  in  Full  Spectrum  Operations  by  TRADOC  (2008).  Soldiers’  interpersonal 
skills  will  largely  determine  their  success  as  followers,  team  members,  and  representatives  of  the 
U.S.  Army  when  deployed.  Once  deployed,  Soldiers  must  shape  the  perceptions  and  win  the 
acceptance  of  the  local  populace,  which  is  possible  through  the  display  of  effective  interpersonal 
skills.  Thus,  there  is  a  need  for  the  Army  to  train  a  broader  range  of  skills  beyond  warrior  skills.  For 
this  reason,  it  is  necessary  to  develop  interpersonal  skills  through  the  use  of  technology  or  other 
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means  that  will  compress  time  (“TRADOC  Pamphlet  525-3-7”,  2008).  To  better  understand  the 
role  of  videogame-based  training  in  training  interpersonal  skills,  this  section  will  review  the 
current  trends  for  interpersonal  skills  training. 

Training  interpersonal  skills  is  not  common  in  the  military,  but  industry  and  education  have 
begun  to  use  different  media  to  implement  this  form  of  training.  The  most  common  forms  of 
training  used  in  interpersonal  skills  training  include  lecture-based  classroom  courses,  leadership 
seminars,  coaching,  scenistic  methods,  and  role-playing.  Maki  and  Maki  (2002)  found  lecture 
courses  were  the  most  commonly  used  training  medium  by  organizations.  Lecture  courses  have 
been  used  to  teach  factual  and  procedural  knowledge  as  well  as  interpersonal  skills.  Formal 
classroom  training  is  often  supplemented  with  post-training  feedback  and  self-coaching  (Tews  & 
Tracey,  2008). 

An  example  of  a  classroom-based  interpersonal  skills  training  is  seen  in  research  conducted  by 
Tews  and  Tracey  (2008).  The  foundation  of  their  research  was  based  on  a  program  conducted  in 
an  organization’s  centralized  corporate  training  center.  The  training  program  was  focused  on 
developing  supervision  skills  by  aiming  to  develop  five  general  skill  sets.  The  general  skills 
were  clearly  communicating  performance  expectations,  observing  employee  performance, 
rewarding  employees  with  frequent  praise  and  recognition,  addressing  performance  issues,  and 
creating  enthusiasm  for  hard  work.  The  interpersonal  skills  training  format  lasted  8  hours  and 
included  lectures,  discussions,  videos,  and  role-playing  activities.  No  formal  evaluation  of  the 
effectiveness  of  this  program  has  been  done.  However,  the  problem  with  lecture  courses  is  that 
although  they  may  be  appropriate  for  achieving  cognitive  objectives,  they  are  hardly  sufficient 
for  psychomotor  or  interpersonal  skills.  Lectures  are  too  passive  a  medium  for  most  adults 
(Pedler,  1978).  Additionally,  lectures  do  not  address  trainee  individual  differences  such  as 
interests,  abilities,  background  and  personalities.  Lectures  prevent  individualized  feedback  and 
reinforcement  from  trainers  (Wexley  &  Latham,  1991).  Furthermore,  some  researchers  argue 
that  formal  classroom  training  produces  individuals  who  are  only  able  to  perform  crude  and 
forced  imitations  of  target  behaviors  (Tews  &  Tracey,  2008). 

Another  popular  trend  in  training  interpersonal  skills  is  through  the  use  of  coaching  (Sparrow, 
2006).  Coaching  has  been  claimed  to  be  especially  well  suited  to  helping  individuals  develop 
their  interpersonal  skills  (Berriman,  2007).  The  coach  observes  the  employee’s  behavior  and 
offers  advice  on  how  to  approach  situations.  More  specifically,  individuals  assigned  as  coaches 
utilize  skills  such  as  active  listening,  purposeful  questioning,  providing  helpful  and  objective 
feedback  (Berriman,  2007),  and  role  modeling  appropriate  behaviors  (Chakrabarty,  Oubre,  & 
Brown,  2008).  Some  drawbacks  of  coaching  are  that  it  is  very  time  intensive,  there  is  focus  on 
only  one  person  at  a  time,  and  little  to  no  evaluation  is  conducted  to  determine  whether 
interpersonal  skills  have  improved.  In  addition,  coaching  has  the  potential  to  cause  stress,  thus 
care  has  to  be  taken  to  decide  which  employee  would  benefit  most  from  such  an  intrusive 
approach  (Berriman,  2007). 

A  newer  approach  being  adopted  to  expand  interpersonal  skills  is  known  as  the  scenistic  method. 
This  approach  involves  the  use  of  incidents,  scenarios,  scripts,  and  episodes  during  the  training. 
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The  goal  of  this  training  approach  is  to  help  learners  practice  skills  such  as  questioning,  making 
assumptions,  defining  issues  and  problems,  problem  finding,  and  problem  solving  (Lyons,  2006). 

Lyons  (2006)  conducted  a  training  evaluation  to  determine  the  effectiveness  of  a  case-based 
modeling  or  type  of  scenistic  method  compared  to  traditional  training.  A  case-based  modeling 
approach  consists  of  eight  steps.  These  steps  are:  (1)  providing  participants  in  written  form  a 
case  situation  or  scenario,  (2)  allowing  participants  to  review  information  for  understanding  and 
any  necessary  clarifications  are  sought  through  small-group  discussions,  (3)  a  discussion  in  an 
open  forum  is  conducted  to  reach  consensus  about  the  critical  issues,  (4)  participants  brainstonn 
potential  interventions  for  treating  the  issues,  (5)  participants  identify  the  several  elements 
necessary  to  address  the  issues,  (6)  rehearsal  of  the  behavioral  script  of  the  job  or  task,  (7) 
develop  qualitative  and  quantitative  evaluations  of  effective,  skillful  intervention  practices  and 
(8)  continue  rehearsing  until  the  participant  is  prepared  to  field  test  the  intervention. 

Once  the  case-based  modeling  was  developed  and  conducted,  participants  were  rated  on  their 
performance  in  comparison  to  a  traditional  group.  Performance  was  assessed  by  a  panel  of 
judges  using  a  behaviorally  anchored  rating  scale.  Participants  met  with  a  panel  of  judges  for  20 
minutes  and  were  asked  to  explain  how  they  would  plan  and  manage  a  sales  call  with  customers. 
Results  of  the  training  evaluation  showed  that  the  case-based  modeling  group  had  higher 
performance.  However,  there  were  several  issues  with  the  design  of  the  research  and  with  this 
form  of  training.  First,  the  different  training  groups  were  trained  by  different  trainers  in  different 
locations  who  were  able  to  design  their  own  curriculum.  Also,  details  of  the  traditional  group 
process  only  included  that  the  trainers  used  traditional  training  approaches.  Furthennore,  an 
issue  with  this  form  of  training  is  that  it  requires  a  lot  of  preliminary  work.  Rating  scales  such  as 
behaviorally  anchored  rating  scales  need  to  be  developed  and  validated  before  they  can  be  used 
during  the  training  (Murphy  &  Cleveland,  1995).  A  lot  of  time  and  resources  are  required  from 
the  organization  to  conduct  this  training. 

Lastly,  the  method  of  role-playing  is  still  very  popular  in  the  interpersonal  skills  training  field. 
Role-playing  is  a  learning  activity  in  which  participants  act  out  a  set  of  defined  role  behaviors.  A 
role-playing  scenario  can  include  mimicking,  demonstration  or  illustration  of  specific  concepts, 
problems  or  situations  (Sogunro,  2004).  A  common  set-up  for  a  role-playing  training  includes 
lectures,  question-and-answer  periods,  small  group  discussions,  case  studies,  structured 
experiences,  and  role-playing.  This  form  of  training  may  require  some  previous  experience  in 
role-playing.  Experience  is  needed  because  the  stage  of  running  the  role  play  may  be  too 
complex  for  beginners  because  they  must  pay  attention  to  verbal  and  nonverbal  aspects  of  the 
conversation  and  at  the  same  time  have  to  control  emotions  and  deal  with  the  pace  of  the 
exchange  (Holsbrink-Engels,  1997).  Furthermore,  because  there  is  little  or  no  time  to  reflect  on 
the  actions  that  occurred  during  the  interaction,  it  is  difficult  for  students  to  keep  track  of  what 
they  learned  and  in  the  order  in  which  it  was  learned.  Novices  may  also  fail  to  build  the  intended 
skills  due  to  a  lack  of  repeated  exposure  (Holsbrink-Engels,  1997).  Researchers  have  also  stated 
that  a  major  issue  with  training  methods  such  as  role  playing  or  discussions  is  that  asking 
inexperienced  trainees  to  work  out  the  interpersonal  skill  together  is  a  potentially  serious 
problem  in  that  the  necessary  concepts  and  skills  that  are  not  effectively  learned  are  probably 
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learned  on  the  job,  where  the  cost  of  mistakes  is  much  greater  (Ross,  Pollman,  Perry,  Welty,  & 
Jones,  2001). 

Overall,  the  previous  training  approaches  for  teaching  interpersonal  skills  have  been  shown  to  be 
time-consuming,  costly,  and  ill-equipped  to  respond  to  the  complexity  of  learning  interpersonal 
skills.  Interpersonal  skills  are  complex  cognitive  skills  (Van  Merrienboer,  Kester,  &  Paas,  2006) 
and  learning  them  may  create  a  high  cognitive  burden.  For  these  reasons,  modem  training 
approaches  must  be  able  to  respond  and  adapt  to  the  different  needs  of  interpersonal  skills 
training. 

Computer-Based  Adaptive  Training  for  Interpersonal  Skills 

In  defining  the  type  of  medium  used  during  training  it  is  common  for  the  literature  to  use 
computer-based  training,  simulations,  games,  and  adaptive  training  interchangeably.  However, 
there  are  some  differences  among  these  media.  Computer-based  training  requires  that  instruction 
be  through  the  use  of  a  computer  terminal  (Wexley  &  Latham,  1991).  A  type  of  computer-based 
training  can  involve  a  game.  A  game  is  an  artificially  constructed,  competitive  activity  with  a 
specific  goal,  a  set  of  rules  and  constraints  that  are  located  in  a  specific  context  (Hays,  2005). 

An  example  of  a  computer-based  game  is  TETRIS,  a  classic  puzzle  game.  In  Tetris  the 
individual  must  change  the  shape  of  falling  blocks  to  complete  horizontal  rows  of  blocks,  which 
disappear  once  completed.  The  objective  is  to  clear  as  many  rows  as  possible  to  receive  points. 

It  is  also  possible  for  a  game  to  be  made  as  a  simulation.  A  simulation  attempts  to  recreate 
relevant  aspects  of  some  topic  or  phenomenon  so  learners  can  interact  with  it  and  observe  the 
consequences  of  their  interactions  (Hays,  2005).  An  example  of  a  simulation  is  the  Combat 
Flight  Simulator,  which  simulates  military  aircraft  and  their  operations.  In  the  simulator,  players 
leam  how  to  fly  an  aircraft  of  their  choice.  Adaptive  training  is  a  form  of  computer  assisted 
instruction  that  automatically  adjusts  stimuli,  problems,  or  tasks  presented  to  trainees  in 
accordance  with  their  performance  level  (Wexley  &  Latham,  1991).  There  are  three  basic 
elements  of  adaptive  training.  First,  performance  is  continually  measured.  Second,  the  problem 
or  task  can  be  changed  in  difficulty  for  the  learner.  Third,  the  scenario  varies  in  relation  to  how  a 
trainee  is  currently  perfonning  (Wexley  &  Latham,  1991).  According  to  the  TRADOC  Pamphlet 
525-3-7  (2008),  adaptive  training  is  a  useful  element  to  include  in  videogame-based  training  for 
the  Army  because  it  increases  tailorability  and  efficiency  of  training.  The  ability  to  adapt 
training  to  the  individual  needs  of  a  specific  Soldier  reduces  the  time  and  cost  to  achieve  training 
objectives.  Furthermore,  the  TRADOC  Pamphlet  525-3-7  (2008),  suggests  that  adaptive  training 
enables  Soldiers  to  learn  ways  that  are  most  effective  for  them  and  that  will  meet  future 
challenges  using  the  most  flexible  solutions  possible.  The  current  research  will  be  examining  and 
using  a  combination  of  these  media.  The  focus  of  this  research  is  computer-based  training,  which 
involves  a  simulation  game  that  can  adapt  to  an  individual’s  responses. 

The  use  of  computer-based  training  began  with  the  idea  of  Programmed  Instruction,  which  was 
influenced  by  B.F.  Skinner  in  the  late  1950s.  This  involved  mostly  rigorously  controlled 
experiments.  This  approach  presented  information,  provided  questions,  and  the  learner  was 
given  feedback  based  on  responses  in  an  attempt  to  share  specific  knowledge  gradually 
(Kamouri,  1983-84).  Programmed  Instruction  was  more  commonly  used  for  the  acquisition  of 
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general  text  information,  specific-task  related  infonnation,  and  job  skills  and  procedures 
(Shoemaker  &  Holt,  1965).  There  were  several  benefits  from  the  use  of  this  medium  such  as 
significant  savings  in  training  time  and  scheduling  flexibility  (Shoemaker  &  Holt,  1965).  Still, 
there  were  serious  limitations  such  as  the  lack  of  motivation  from  the  trainee,  the  need  for 
individualized  feedback,  limited  applications,  expensive  and  large  equipment,  and  the  need  for 
technically  competent  personnel  that  were  not  readily  available  (Kamouri,  1983-84).  This 
approach  was  then  followed  by  a  technological  boom  that  supported  the  spread  of  technology- 
driven  training.  This  boom  expanded  the  field  in  terms  of  the  type  of  research  and  function  of 
computer-based  training. 

The  next  approach  towards  computer-based  training  stemmed  from  human  factors  research.  In 
this  field,  researchers  examined  the  factors  affecting  human  perfonnance  in  the  context  of  man- 
machine  interaction.  Furthermore,  researchers  and  practitioners  studied  the  various  aspects  of  a 
system’s  interface  such  as  hardware,  administration  of  training,  and  software  development 
(Kamouri,  1983-84).  The  growth  of  technology-driven  training  was  further  accelerated  with  the 
inclusion  of  learning  theory  and  individual  differences  considerations.  Learning  theory  and  the 
study  of  individual  differences  were  incorporated  into  the  program  design  of  computer-based 
training  to  enhance  training  performance  (Kamouri,  1983-84).  Cognitive  theory  researchers  also 
played  a  role  in  the  development  of  the  current  computer-based  training  systems.  The  influence 
of  this  field  was  acknowledging  the  importance  of  complex  information  processing  (Goldstein, 
1980).  Cognitive  theory  was  being  incorporated  into  computer  assisted  instruction  programs  in 
education  (Block,  1979).  Now  computer-based  programs  train  the  operation  of  complex 
equipment  and  assist  in  developing  highly  specialized  skills  such  as  flight  procedures  and 
engineering  principles  (Kamouri,  1983-84). 

As  the  cost  of  technology  continues  to  decline  and  the  cost  of  face-to-face  training  increases, 
more  organizations  are  beginning  to  implement  computer-based  training  applications  such  as 
simulations,  videogames,  Web-based  training,  and  distance  learning  (Bell  &  Kozlowski,  2002)  to 
help  combat  the  issues  found  with  traditional  training  methods.  Research  supports  the  notion 
that  superior  and  more  rapid  learning  is  associated  with  more  personalized,  self-paced,  and  self- 
directed  computerized  instruction  especially  when  training  adults  (Kamouri,  1983-84). 

The  key  to  computer-based  adaptive  training  for  interpersonal  skills  is  its  ability  to  overcome 
shortcomings  of  other  methods.  For  example,  computer-based  adaptive  training  can 
accommodate  individual  learning  differences.  Some  trainees  require  more  time  or  more  sessions 
to  better  acquire  the  necessary  skills  and  concepts.  However,  the  fonnat  of  traditional  training 
techniques  and  the  high  cost  do  not  always  make  this  possible.  Computer-based  adaptive 
training  simulates  work  experiences  in  a  setting  where  trainees  can  experiment  with  new 
behavioral  styles,  and  it  has  the  capability  of  teaching  as  many  trainees  with  as  many  sessions 
needed.  This  is  an  important  feature  because  the  direct  experience  of  a  learning  activity  is  key  to 
bringing  about  real  understanding  and  desired  change  in  people  (Sogunro,  2004).  This  form  of 
training  might  better  help  trainees  learn  to  recognize  important  decision  points  and  apply  the 
theory  when  making  decisions  (Ross  et  ah,  2001).  Furthermore,  computer-based  adaptive 
training  can  provide  individualized  immediate  feedback  for  each  trainee. 
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Another  important  feature  needed  in  interpersonal  skills  training  is  the  ability  to  reflect  on  one’s 
actions  and  performance  and  to  plan  ahead  (Holsbrink-Engels,  1997).  Traditional  interpersonal 
skills  training  does  not  always  include  this  ability,  however.  For  example,  role-playing  requires 
constant  interaction.  On  the  other  hand,  computer-based  adaptive  training  provides  opportunities 
to  control  what  is  learned  and  record-keeping  facilities  for  monitoring  and  recording  the 
contributions  of  individual  trainees  (Holsbrink-Engels,  1997).  It  is  a  technique  that  provides  the 
trainee  with  an  opportunity  for  active  participation,  trying  out  new  behaviors  in  a  safe 
environment,  feedback,  and  practice  (Wexley  &  Latham,  1991).  Computer-based  adaptive 
training  also  solves  the  issue  of  having  an  inexperienced  trainer  (Crosbie,  2005). 

One  popular  form  of  computer-based  training  is  videogames.  The  use  of  videogames  has 
become  a  popular  medium  of  training  because  some  research  shows  that  games  can  be  effective 
tools  for  learning  and  understanding  complex  subject  matter  (Ricci,  Salas,  &  Cannon-Bowers, 
1996).  Videogames  have  also  been  found  to  be  immersive  and  engaging  in  ways  that  traditional 
workbooks  or  manuals  are  not  (Garris,  Ahlers  &  Driskell,  2002).  Another  reason  for  the 
widespread  interest  among  practitioners  is  that  research  has  shown  that  skills  learned  in 
videogame -based  training  environments  do  transfer  to  real-life  situations  (Gopher,  Weil,  & 
Bareket,  1994;  Knerr,  Simutis,  &  Johnson,  1979).  Thus,  research  supports  the  notion  that  people 
can  learn  from  videogames. 

Although  research  shows  that  videogames  can  be  an  effective  tool  for  training,  most  of  the  tasks 
trained  have  been  math,  electronics,  and  economics  (Hays,  2005).  A  comprehensive  literature 
review  on  videogames  that  included  over  270  documents  was  conducted  by  Hays  (2005).  In  this 
literature  review  only  48  of  the  studies  provided  empirical  data  of  the  instructional  effectiveness 
of  games  and  there  were  no  studies  that  included  training  interpersonal  skills.  The  review  did 
find  that  videogames  that  were  embedded  in  instructional  programs  and  included  performance 
evaluation,  debriefing,  and  feedback  were  effective  in  enhancing  cognitive  learning  (Hays, 

2005).  A  previous  study  reviewing  business  games  used  to  teach  strategic  management  skills 
found  more  positive  results  (Wolfe,  1997).  The  review  was  conducted  on  studies  between  1966 
and  1988,  and  found  evidence  that  computer-based  general  management  games  are  effective  in 
producing  knowledge  gains.  Wolfe  (1997)  also  found  that  the  only  alternative  instructional 
approach  that  was  similar  to  games  was  case  studies.  Although  both  methods  produced  learning, 
the  games  appeared  to  be  superior  (Wolfe,  1997).  Similarly,  Whitehall  and  McDonald  (1993) 
and  Ricci  et  al.  (1996)  found  that  training  incorporating  game  features  leads  to  improved 
learning.  Thus,  there  is  potential  for  videogames  to  be  used  for  training  interpersonal  skills. 

There  has  been  some  recent  movement  in  the  development  of  videogame-based  interpersonal 
skills  training.  For  example,  ELECT  BILAT  (Hall  et  al.,  2006)  is  a  videogame-based  simulation 
used  for  Soldiers  to  practice  conducting  bilateral  engagements  in  a  cultural  context.  In  BILAT, 
the  trainees  have  meetings  concerning  the  cultural  context,  gather  information,  conduct  meetings 
and  negotiate  whenever  necessary.  Additionally,  the  participants  are  given  targeted  feedback  by 
a  coach.  Although  this  training  tool  shows  potential,  there  has  been  no  formal  evaluation  to 
determine  whether  learning  occurs. 
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Another  videogame-based  interpersonal  skills  training  used  and  evaluated  is  the  McGill 
Negotiation  Simulator  designed  to  teach  negotiation  skills  during  a  sales  negotiation  scenario 
(Ross  et  ah,  2001).  The  simulator  allowed  for  the  trainee  to  choose  from  several  responses 
during  a  negotiation  scenario.  Then,  the  preprogrammed  opponent  responds  in  a  way  that  is 
consistent  with  the  underlying  negotiation  theory.  The  researchers  found  that  with  the  use  of  a 
computer-based  adaptive  interpersonal  skills  videogame,  trainees  demonstrated  a  significant 
increase  in  learning  relative  to  a  control  group.  Not  only  did  the  students  learn  more  compared 
to  a  control  group  but  they  also  enjoyed  the  simulator.  Similar  results  were  found  in  their 
replication  of  the  study  (Ross  et  al.,  2001).  Additional  research  evaluating  computer-based 
adaptive  interpersonal  skills  training  program  was  done  by  Torney-Purta  (1998)  for  the  ICONS 
Computer-Assisted  Simulation.  The  ICONS  simulation  was  designed  to  teach  international 
concepts,  content,  and  negotiation  strategies  and  skills.  The  simulation  combined  a  lecture 
format  with  a  small-group  “seminar”  type  learning  environment.  The  small  group  discussions 
were  used  to  prepare  students  for  participation  in  the  negotiation  simulation.  Participants 
negotiated  with  peers  at  overseas  institutions.  The  study  found  that  later  in  the  semester  the 
students  showed  more  elaborate  thinking  in  dealing  with  international  issues  and  seeing  linkages 
between  problems.  The  evaluation  also  showed  that  not  only  were  students  able  to  negotiate  in 
an  international  setting  but  they  were  thoughtful  about  their  reasons  for  choosing  a  specific 
negotiation  style.  Lastly,  the  majority  of  students  became  highly  involved  in  the  program  and 
valued  learning  using  the  computer  technology. 

Another  computer-based  simulation  that  trains  interpersonal  skills  was  developed  by  a 
telecommunications  firm  with  the  goal  of  encouraging  managers  to  take  more  responsibility  for 
leadership,  and  to  increase  managers’  understanding  and  knowledge  of  the  industry  (Drew  & 
Davidson,  1993).  The  computer-based  game  was  made  up  of  four  teams  of  usually  four 
participants  who  would  make  decisions  on  new  products,  marketing,  sales  force,  customer 
service,  and  operations.  Each  team  member  was  assigned  a  role  such  as  VP  of  marketing  and  VP 
of  product  introduction.  The  game  required  each  team  to  search  for  data  and  decide  about  issues 
such  as  new  products,  pricing,  and  budget  allocations  between  departments.  The  game  required 
participants  to  work  together  and  be  able  to  come  to  consensus  on  many  decisions.  Furthermore, 
the  simulation  provided  an  opportunity  to  share  experiences  and  discussions.  At  the  end  of  the 
simulation,  participants  received  feedback  on  their  behavior. 

The  previous  discussion  has  shown  that  computer-based  adaptive  training  for  interpersonal  skills 
is  still  a  growing  and  emerging  field  as  practitioners  and  researchers  alike  have  recognized  its 
benefits.  Although  minimal  research  has  been  conducted  in  this  area,  studies  have  shown  that 
videogame  role-playing  enhances  interpersonal  skills  development  by  providing  practice  in  a 
conversational  scenario,  offering  opportunities  for  reflection,  and  capturing  individual 
contribution  and  learning  (Holsbrink-Engels,  1997). 

Although  preliminary  research  has  found  potential  for  the  use  of  videogames,  simulations,  and 
computer-based  adaptive  training  for  teaching  interpersonal  skills  there  are  still  some  issues  with 
this  form  of  training.  First,  expecting  trainees  to  learn  from  trial  and  error  with  the  virtual  human 
is  problematic  because  errors  may  go  unrecognized.  Trainees  may  come  to  believe  they  were 
successful  when  in  fact  there  was  a  problem  with  their  response  that  deserved  attention  (Core  et 
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ah,  2007).  Second,  advanced  technologies  offer  flexibility,  customization,  and  speed  but  it  is 
important  to  ensure  that  the  instructional  integrity  of  these  programs  is  maintained  and  evaluated. 
There  is  still  little  consensus  in  the  research  on  game  features  that  support  learning  (Garris, 
Ahlers,  &  Driskell,  2002).  Researchers  have  also  suggested  that  simulators  are  ineffective 
because  they  do  not  directly  measure  the  instructional  objectives  of  the  training  or  ensure  that  the 
learner  has  met  these  objectives  (Hays,  2005).  In  his  review,  Hays  (2005)  stated  that  the  current 
instructional  games  literature  is  made  up  of  ill-defined  terms  and  methodological  flaws.  To  help 
resolve  these  issues,  the  current  literature  needs  to  examine  a  fundamental  element  of  training: 
feedback.  Feedback  is  information  concerning  one’s  performance.  Feedback  is  necessary  for 
correcting  mistakes  and  learning.  Nevertheless,  the  computer-based  adaptive  training  literature 
rarely  discusses  the  use  of  feedback  even  though  most  programs  include  some  form  of  feedback. 
Furthermore,  research  has  not  compared  which  types  of  feedback  are  the  most  effective  in 
teaching  interpersonal  skills  in  a  computer-based  adaptive  training  environment.  The  present 
research  seeks  to  address  this  gap  in  the  literature  by  testing  competing  feedback  theories  to 
determine  the  most  effective  instructional  design  for  training  interpersonal  skills. 

Feedback  and  Training  Design 

The  primary  goal  of  training  is  to  improve  performance  and  to  achieve  this  it  is  critical  for 
trainees  to  be  provided  with  feedback.  Feedback  is  information  given  by  an  external  agent 
regarding  some  aspect  of  one’s  task  perfonnance  (Kluger  &  DeNisi,  1996).  Feedback  is 
essential  in  training  because  it  serves  three  major  functions  in  promoting  learning  and 
motivation.  First,  it  tells  trainees  whether  their  perfonnance  is  correct  thus  allowing  them  to 
make  any  necessary  corrections  to  their  subsequent  behavior.  Second,  feedback  makes  the 
learning  process  more  interesting,  which  increases  a  trainee’s  willingness  to  leam,  and  third, 
feedback  leads  to  the  setting  of  goals  to  improve  perfonnance  (Wexley  &  Latham,  1991). 
Training  researchers  suggest  providing  trainees  with  debriefing  and  feedback  that  clearly  states 
how  their  behaviors  helped  them  meet  the  instructional  objectives  (Hays,  2005).  The  feedback 
provided  to  the  trainee  should  result  in  the  development  of  better  mental  models  that  will  more 
likely  be  transfened  to  applied  settings  (Heimbreck,  Frese,  Sonnentag,  &  Keight,  2003). 
Furthermore,  research  shows  that  training  characteristics  such  as  practice,  behavior  modeling, 
part-  versus  whole-task  learning,  and  feedback  are  positively  related  to  performance  (Alvarez, 
Salas  &  Garofano,  2004).  Regardless  of  these  suggestions  and  findings,  few  studies  have 
directly  looked  at  the  impact  of  feedback  in  training.  Feedback  in  training  is  a  commonly 
considered  training  design  feature,  as  are  learning  principles  such  as  massed  versus  spaced 
practice,  but  feedback  is  rarely  examined  for  its  own  unique  contribution  to  the  instructional 
design  process.  Even  less  is  known  about  feedback  in  a  computer-based  adaptive  training 
environment.  There  has  been  some  research  examining  the  differences  between  face-to-face 
feedback  versus  computer  mediated  feedback  (Earley,  1988;  Kluger  &  Adler,  1993;  Kluger  & 
DeNisi,  1996;  Markus,  1994b;  Sproull  &  Kiesler,  1986).  Generally,  studies  have  found  that 
people  prefer  to  seek  or  receive  feedback  from  computers  rather  than  face-to-face.  However, 
exactly  what  kind  of  feedback  and  how  often  it  is  given  is  rarely  examined  in  the  computer 
mediated  feedback  literature.  For  this  reason,  the  current  effort  will  attempt  to  close  this  gap  by 
examining  the  effect  feedback  frequency  and  feedback  sign  will  have  in  a  computer-based 
adaptive  training  environment. 
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Feedback  Frequency 

Feedback  frequency  refers  to  the  number  of  times  a  feedback  intervention  is  provided  during 
training.  In  this  investigation,  “frequent”  feedback  is  defined  as  feedback  given  to  an  individual 
immediately  at  the  end  of  each  interaction  within  the  mission  and  again  at  the  end  of  the  mission. 
“Infrequent”  feedback  is  given  once  per  mission,  at  the  end.  Several  conflicting  theories  have 
been  proposed  to  help  predict  an  individual’s  performance  based  on  the  frequency  of  the 
feedback  they  received.  According  to  the  feedback  intervention  theory,  frequent  feedback  is 
considered  detrimental  to  the  acquisition  of  a  complex  skill.  On  the  other  hand,  the  ACT-R 
model  suggests  that  frequent  and  immediate  feedback  is  essential  for  acquiring  any  skill.  These 
competing  theories  are  discussed  in  more  detail  below. 

Feedback  intervention  theory.  In  most  cases,  feedback  has  been  found  to  accomplish  its 
intended  purpose,  which  is  to  provide  information  about  current  job  performance  to  facilitate  and 
motivate  performance  improvements;  however,  there  have  been  instances  where  this  does  not 
occur.  Kluger  and  DeNisi  (1996)  noticed  these  inconsistencies  and  conducted  a  meta-analysis  on 
the  effect  of  feedback  interventions  on  performance.  The  authors  found  that  in  a  third  of  the 
studies  feedback  decreased  or  had  no  effect  on  performance.  Not  much  has  changed  in  the 
feedback  literature  since  this  meta-analysis.  There  is  still  a  debate  regarding  whether  feedback 
improves  performance  (London,  2003)  and  under  what  conditions.  To  help  clarify  these 
contradictions  Kluger  and  DeNisi  (1996)  proposed  the  feedback  intervention  theory. 

The  feedback  intervention  theory  assumes  that  behavior  is  goal-directed  and  is  based  on  five 
assumptions:  (1)  behavior  is  regulated  by  a  comparison  of  feedback  with  a  goal  or  standard,  (2) 
these  goals  are  ordered  hierarchically,  (3)  attention  is  limited  so  only  feedback  that  demonstrates 
a  gap  between  the  current  level  of  perfonnance  and  the  goal  will  regulate  behavior,  (4)  attention 
is  normally  directed  to  a  moderate  level  in  the  hierarchy,  and  (5)  feedback  interventions  can  alter 
the  location  of  attention  and  in  doing  so  can  affect  behavior.  Given  that  these  assumptions  are 
true,  the  framework  of  the  theory  consists  of  a  three-level  hierarchy  where  feedback  may  focus 
the  individual’s  attention  at  any  of  these  levels.  When  feedback  directs  attention  away  from  the 
task  at  hand  to  factors  associated  with  the  self  or  ego  it  is  viewed  as  working  at  the  highest  level 
in  the  hierarchy,  the  self  or  meta-task  level.  The  middle  or  moderate  level  is  known  as  the 
motivation  level.  Feedback  directed  at  the  motivation  level  provides  an  indication  of  how  well 
individuals  are  meeting  their  goals  and  presumably  will  inspire  motivation  and  persistence.  The 
lowest  level  is  the  task  learning  level,  which  occurs  when  the  attention  is  brought  to  the  specific 
details  of  the  task.  This  occurs  when  working  harder  (task  motivation  level)  does  not  lead  to  the 
desired  perfonnance  level  and  the  individual  must  focus  on  specific  processes  or  strategies  to 
improve  performance. 

This  theory  states  that  the  moderate  or  task  motivation  level  is  where  attention  is  normally 
directed.  The  goals  at  this  level  are  related  to  actual  task  performance.  At  the  task  motivation 
level  the  primary  focus  is  on  the  task  itself  and  working  to  reduce  the  discrepancy  between  the 
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desired  goals  and  the  current  state.  This,  in  essence,  is  the  motivational  (persistence  and 
direction)  impact  of  feedback.  It  is  believed  that  most  feedback  interventions  are  focused  at  this 
level  and  only  when  the  person  is  not  able  to  reduce  the  discrepancy  does  the  attention  shift  up 
(meta-task  level)  or  down  the  hierarchy  (task  learning  level).  Furthermore,  interventions  that 
focus  attention  at  this  level  are  the  most  likely  to  produce  the  desired  effect  on  performance 
(DeNisi  &  Kluger,  2000). 

The  highest  level  of  the  hierarchy  refers  to  the  self-level  where  the  goals  relate  to  an  individual’s 
self-concept.  Feedback  interventions  focus  attention  at  this  level  when  success  at  the  task  is 
equated  with  some  higher  order  goal  or  when  performance  is  central  to  our  self-concept  (DeNisi 
&  Kluger,  2000).  At  this  level  it  is  unlikely  that  the  individual  will  abandon  goals  but  instead 
concern  over  reasserting  or  defending  one’s  self-image  could  interfere  with  the  ability  to  focus 
on  the  task  and  thus  decrease  performance  (DeNisi  &  Kluger,  2000). 

Lastly,  the  lowest  level  is  the  task  learning  level  where  the  attention  is  brought  to  the  details  of 
the  task  or  the  actions  involved  in  perfonning  the  task.  If  attention  shifts  to  the  task  learning 
level  the  concern  may  turn  to  the  tone  of  voice  one  used  to  speak  to  someone  rather  than  the 
approach  used  to  explain  a  concept.  Thus,  if  feedback  causes  a  person  to  focus  too  much  on  the 
details  of  the  process  it  may  cause  perfonnance  to  decrease. 

The  feedback  intervention  theory  helps  explain  how  feedback  frequency,  or  how  often  feedback 
is  provided  by  a  source,  can  influence  behavior.  Although  it  is  generally  believed  that  the  more 
frequent  the  feedback,  the  better  (Ilgen,  Fisher,  &  Taylor,  1979),  this  theoretical  framework 
claims  that  this  may  not  always  be  the  case.  In  their  meta-analysis,  Kluger  and  DeNisi  (1996) 
found  that  feedback  interventions  associated  with  complex  tasks  were  more  likely  to  result  in 
declines  in  perfonnance.  A  decline  in  perfonnance  may  occur  because  during  a  complex  task, 
frequent  feedback  diverts  attention  to  the  task  learning  level  and  performance  will  decline  if  that 
feedback  does  not  provide  sufficient  infonnation  needed  to  develop  specific  strategies  for 
improvement.  Thus,  the  feedback  would  serve  as  a  distraction  from  the  task  at  hand  during  a 
time  when  the  person’s  full  attention  is  needed  (DeNisi  &  Kluger,  2005). 

The  concept  of  too  much  feedback  being  a  distraction  is  also  supported  by  the  notion  of 
performance  feedback  overload.  Performance  feedback  overload  refers  to  “a  perception  by  an 
individual  that  he  or  she  has  so  much  accessible  perfonnance  feedback  that  it  is  not  longer 
possible  to  use  it  effectively”  (Salvati,  Gosselin,  &  Morin,  2003).  The  theoretical  ground  for  this 
concept  is  based  on  the  notion  that  individuals  have  limited  information  processing  capabilities. 
Salvati  et  al.  (2003)  identified  several  antecedents  and  consequences  to  feedback  overload. 
Antecedents  included  feedback  value  and  job  experience  while  important  consequences  were  the 
desire  to  respond  to  feedback,  stress,  and  job  performance.  The  authors  suggest  that  the  well- 
accepted  premise  that  more  feedback  will  help  individuals  is  not  necessarily  true  and  instead  too 
much  feedback  may  have  detrimental  effects  on  an  individual. 

There  is  research  that  shows  support  for  the  view  that  frequent  feedback  can  be  detrimental  to 
performance  during  a  complex  skill  (Buchwals  &  Meager,  1974;  Chokkar  &  Wallin,  1984;  Ilgen 
et  al.,  1979;  Janelle,  Barba,  Frehlich,  Tennant,  &  Cauraugh,  1997;  Renn  &  Prien,  1995;  Swinnen 
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et  al.,  1990).  Ilgen  et  al.  (1979)  explain  that  when  learning  a  complex  skill,  such  as  an 
interpersonal  skill,  the  relationships  between  events  are  not  perfectly  correlated  as  they  are  when 
an  individual  is  being  trained  to  estimate  distances  between  objects.  Because  the  relationship  is 
not  perfectly  correlated,  feedback  after  each  trial  can  be  very  misleading  and  detrimental  to 
learning  the  relationship.  Thus,  the  authors  suggest  caution  when  recommending  the  use  of 
frequent  feedback  especially  when  individuals  must  learn  complex  skills.  This  has  been 
demonstrated  in  skill  acquisition  laboratory  studies  (Swinnen  et  al.,  1990)  and  field  studies 
where  more  frequent  feedback  did  not  lead  to  greater  performance  (Chhokar  &  Wallin,  1984). 
Similarly,  Buchwals  and  Meager  (1974)  found  that  delayed  feedback  improved  performance  if 
individuals  remembered  the  original  response.  That  is,  if  the  activities  between  the  response  and 
the  delayed  feedback  did  not  interrupt  the  individual’s  ability  to  recall  information,  then  the 
delayed  feedback  had  no  detrimental  effects  on  performance. 

In  another  study,  Janelle  et  al.  (1997)  examined  whether  participants  who  could  control  the 
amount  of  feedback  they  received  would  perfonn  differently  than  those  who  had  a  rigid  feedback 
schedule  or  those  who  received  more  feedback  while  learning  a  complex  task.  Results  showed 
that  when  participants  were  given  the  opportunity  to  control  the  amount  of  feedback  they 
received  they  asked  for  less  feedback  or  infrequent  feedback  to  learn  their  skills  at  a  level  equal 
to  or  greater  than  groups  not  provided  that  option.  Infrequent  feedback  had  the  same  results  for  a 
study  conducted  by  Renn  and  Prien  (1995).  Researchers  examined  employee  responses  to 
performance  feedback  from  the  task  and  found  that  more  frequent  performance  feedback  may 
not  always  be  associated  with  higher  performance. 

Similarly,  studies  in  the  error-detection  field  have  found  that  compared  to  delayed  feedback, 
frequent  feedback  can  be  detrimental  to  performance.  Swinnen  et  al.  (1990)  found  that  frequent 
knowledge  of  results  degraded  learning  as  measured  on  a  delayed  retention  task.  This  may  occur 
because  a  short  time  lag  between  performance  and  knowledge  of  results  may  degrade  the 
acquisition  of  error-detection  capabilities. 

Overall,  past  research  has  shown  that  frequent  feedback  may  bring  attention  to  the  task  details 
and  in  turn  cause  an  unnecessary  distraction  while  learning  a  complex  or  interpersonal  skill. 

This  may  be  especially  true  in  a  computer-based  adaptive  environment  where  the  use  of 
advanced  technologies  may  require  additional  attention  while  learning  a  complex  skill.  In  line 
with  the  previous  studies  the  following  hypothesis  is  proposed: 

Hypothesis  1:  Following  the  feedback  intervention  theory,  trainees  in  the  infrequent 

feedback  group  will  have  a  higher  change  in  performance  for  the  pre-  and  post-test  and 

Mission  1  and  Mission  2  scores  than  trainees  in  the  frequent  feedback  group. 

Feedback  in  ACT-R  theory.  Anderson’s  ACT-R  theory  (1983;  1996;  2000)  consists  of  a 
detailed  theory  of  skill  acquisition.  According  to  the  theory,  skill  acquisition  occurs  in  three 
phases:  interpretive,  procedural,  and  automatization.  In  the  interpretive  stage,  individuals  learn 
the  necessary  declarative  knowledge  and  act  upon  general  problem-solving  strategies.  The 
procedural  stage  is  when  individuals  form  procedures  based  on  the  compilation  of  the  specific 
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episodes  they  encounter.  Lastly,  the  automatization  stage  refers  to  the  point  where  the  learned 
information  and  compiled  procedures  are  strengthened  and  thus  apply  automatically. 

More  specifically,  the  ACT-R  model  of  skill  acquisition  consists  of  a  hierarchy  of  goals  related 
to  the  learning  process.  The  theory  suggests  that  to  encourage  the  learner  to  solve  problems,  a 
goal  structure  should  be  used  that  will  be  useful  for  later  learning  and  perfonnance.  For 
example,  a  task  can  be  decomposed  into  hierarchical  goals  and  eventually  actions  that  will  form 
an  outline  for  subsequent  behavior.  In  essence,  tasks  should  be  broken  down  to  their  elemental 
components  to  prevent  incorrect  or  unnecessary  steps  being  formed. 

Another  important  element  of  the  ACT-R  model  is  the  need  for  the  learner  to  actually  perform 
the  task  rather  than  just  being  told  about  the  procedure.  Because  learning  takes  place  when  there 
is  previous  behavior  that  can  be  compiled  upon,  simply  having  the  declarative  knowledge  will 
not  be  enough  for  the  skill  to  be  acquired.  Thus,  perfonning  the  tasks  allows  for  integration  of 
the  material. 

Therefore,  the  ACT-R  theory  suggests  several  guidelines  for  the  design  and  use  of  feedback.  For 
the  initial  stages  of  skill  acquisition,  feedback  should  include  an  indication  of  the  goal  structure 
of  the  performance.  When  there  is  an  error  in  the  learner’s  performance  the  feedback  should 
contain  the  correct  information,  including  any  necessary  steps  needed  to  accomplish  the  goals. 
Additionally,  feedback  should  take  place  immediately  after  the  error  has  occurred  and  be  placed 
within  the  context  of  the  problem.  Delayed  feedback  slows  down  the  process  of  compiling 
correct  infonnation  and  can  result  in  learning  procedures  that  are  incorrect  and  then  later  applied 
to  future  problems. 

Most  of  the  research  based  on  the  ACT-R  model  has  centered  around  “hard  skills”  or  well- 
defined  domains  such  as  science,  geometry,  mathematics,  and  programming  (Anderson,  Boyle, 

&  Yost,  1985;  McKendree,  1990).  However,  its  application  for  complex  or  ill-defined  situations 
has  been  examined  (Connelly,  2001;  Mascha,  2001)  and  other  studies  have  shown  support  for  the 
need  for  immediate  and  frequent  feedback  in  learning  new  skills  (Anderson,  Kulhavy,  &  Andre, 
1971;  Cook,  1968;  Ivancevich,  Donnelly,  &  Lyon,  1970). 

Anderson  et  al.  (1971)  conducted  a  study  using  a  computer-based  instructional  system  and  found 
that  subjects  who  received  feedback  immediately  after  they  responded  learned  significantly  more 
than  subjects  who  received  delayed  feedback.  Similarly,  research  has  found  that  the  frequency 
of  feedback  given  to  managers  is  directly  related  to  their  attitudes  and  performance.  In  Cook’s 
(1968)  study,  frequent  feedback  was  defined  as  feedback  received  in  the  form  of  quarterly 
reports  and  delayed  feedback  was  feedback  in  the  form  of  yearly  reports.  The  results  indicate 
that  managers  given  frequent  feedback  in  the  form  of  quarterly  reports  had  higher  performance 
and  attitude  ratings  than  those  given  delayed  feedback  in  the  form  of  yearly  reports. 

Furthermore,  contrary  to  the  feedback  intervention  theory,  McKendree  (1990)  found  that  more 
complex  or  ambiguous  tasks  require  a  greater  degree  of  feedback  that  provides  detailed 
information  about  one’s  performance  than  more  well-defined  domains  such  as  science  and 
geometry. 
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Mascha  (2001)  also  examined  the  effects  of  task  complexity  and  feedback  types  on  knowledge 
acquisition.  Results  show  that  participants  receiving  either  type  of  procedural  knowledge 
feedback  performed  better  than  those  who  received  no  feedback.  This  lends  to  the  support  of  the 
ACT-R  prediction  of  a  positive  relationship  between  feedback  or  procedural  cues  and  outcomes. 

The  previous  studies  contradict  the  notion  that  frequent  feedback  is  detrimental  to  learning  and 
instead  provide  support  for  the  contention  that  frequent  feedback  can  be  beneficial  for  teaching 
various  types  of  skills.  According  to  the  ACT-R  model,  trainees  learning  interpersonal  skills 
through  a  computer-based  adaptive  environment  should  be  given  frequent  feedback  throughout 
the  videogame.  Frequent  feedback  will  allow  constant  revision  and  learning  during  the 
interactions  in  the  videogame.  Thus,  the  following  hypothesis  will  be  examined: 

Hypothesis  2:  Following  the  ACT-R  theory  of  skill  acquisition,  trainees  in  the  frequent 

feedback  group  will  have  a  higher  change  in  performance  for  the  pre-  and  post-test  and 

Mission  1  and  Mission  2  scores  than  trainees  in  the  infrequent  feedback  group. 

Feedback  Sign 

Feedback  is  widely  used  in  modern  organizations  to  develop  and  train  employees.  However,  the 
effectiveness  of  positive  versus  negative  feedback  is  still  in  question.  Positive  feedback  indicates 
that  the  standard  (goal)  has  been  met  or  exceeded.  Negative  feedback  indicates  a  discrepancy 
between  the  standard  (goal)  and  performance  that  indicates  the  goal  has  not  been  attained  (Ilies, 
Pater,  &  Judge,  2007).  The  decision  whether  negative  or  positive  feedback  should  be  used 
seems  simple  but  the  current  literature  is  ill-equipped  to  answer  it,  particularly  within  a  formal 
training  context.  Similar  to  the  feedback  frequency  literature,  there  are  theories  explaining  the 
relationship  between  feedback  sign  and  performance  that  directly  oppose  each  other.  Social 
cognitive  theory  (Bandura,  1986)  helps  explain  the  area  of  research  that  shows  positive  feedback 
enhances  performance.  Contrary  to  this  is  the  control  theory  perspective  (Carver  &  Scheier, 
1998),  which  supports  the  notion  of  negative  feedback  enhancing  performance.  This  effort  will 
attempt  to  clarify  the  contradictions  in  the  literature  by  testing  the  two  competing  theories,  which 
are  discussed  in  more  detail  below. 

Social  cognitive  theory.  Social  cognitive  theory  is  a  learning  theory  based  on  the  idea  that 
people  leam  by  observing  others,  people  behave  in  certain  ways  to  reach  goals  and  behavior  is 
motivated  and  regulated  by  one’s  cognitions  (Bandura,  1986).  One  important  set  of  cognitions  is 
self-efficacy,  or  judgments  of  how  well  one  can  accomplish  a  task  at  designated  levels.  Efficacy 
judgments  are  task-specific  and  whether  accurate  or  faulty  they  influence  behavior  by 
determining  task  choices,  effort,  and  persistence  in  the  face  of  obstacles.  The  belief  is  that  the 
higher  the  level  of  self-efficacy,  the  higher  the  performance  accomplishments. 

According  to  self-efficacy  theory,  people  avoid  activities  that  they  believe  are  beyond  their 
coping  capabilities,  but  they  undertake  and  perform  activities  they  judge  themselves  capable  of 
managing.  Therefore,  receiving  positive  feedback  may  motivate  individuals  to  continue 
increasing  their  performance  to  reach  their  goals  because  it  is  an  indication  of  success  and  that 
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the  ability  to  accomplish  the  task  is  present.  On  the  other  hand,  negative  feedback  may  cause 
avoidance  of  the  task  because  it  causes  one  to  believe  the  task  is  beyond  one’s  capability. 

There  is  research  supporting  social  cognitive  theory;  however,  not  many  studies  have  focused  on 
applying  this  theory  to  training  interpersonal  skills.  Regardless,  research  suggests  positive 
feedback  is  necessary  to  enhance  performance  (Becker  &  Klimoski,  1989;  Chakrabarty,  Oubre, 

&  Brown,  2008;  Ilgen,  1979;  Jaworski  &  Kohli,  1991;  Kluger  &  DeNisi,  2005;  Martocchio  & 
Webster,  1992;  Shrauger  &  Rosenberg,  1970;  Van-Dijk  &  Kluger,  2004)  and  negative  feedback 
can  have  detrimental  effects  (Ilgen  &  Davis,  2000;  Waldersee  &  Luthans,  1994)  or  no  effect 
(Jaworski  &  Kohli,  1991)  on  perfonnance. 

There  is  a  general  conclusion  that  positive  feedback  is  more  pleasant  and  may  enhance  one’s 
self-image  and  thus  it  tends  to  be  recalled  and  perceived  more  accurately  than  negative  feedback 
(Ilgen  et  ah,  1979;  Shrauger  &  Rosenberg,  1970).  It  has  also  been  suggested  that  feedback  may 
affect  perfonnance  through  the  behavioral  reward  properties  inherent  in  positive  feedback 
(Waldersee  &  Luthans,  1994).  According  to  the  behavioral  paradigm,  because  receiving  positive 
feedback  is  considered  desirable  it  is  a  positive  reinforcer. 

In  line  with  this  notion,  Martochio  and  Webster  (1992)  examined  perfonnance  feedback  and  self- 
efficacy  reactions.  The  authors  found  that  positive  feedback  resulted  in  higher  test  performance 
than  negative  feedback.  Furthermore,  positive  feedback  increased  software  self-efficacy  beliefs 
where  negative  feedback  decreased  software  efficacy  beliefs.  Similarly,  positive  feedback  was 
also  found  to  indirectly  influence  performance  in  a  field  experiment  examining  the  main  and 
interactive  effects  of  feedback  and  self-efficacy  on  performance.  Karl,  O’Leary-Kelly  & 
Martocchio  (1993)  found  that  the  more  positive  the  performance  feedback  received,  the  greater 
the  increase  in  individual  self-efficacy.  These  studies  lend  support  for  the  notion  that  social 
cognitive  theory  provides  insight  into  the  mechanisms  by  which  feedback  influences  learning 
and  perfonnance. 

Further  supporting  the  use  of  positive  feedback,  Becker  and  Kilmoski  (1989)  examined  the 
relationship  between  the  perceived  organizational  feedback  environment  and  performance.  The 
researchers  found  that  positive  feedback  from  a  supervisor  related  to  higher  performance 
whereas  negative  feedback  from  the  supervisor  related  to  lower  performance.  Positive  feedback 
has  also  been  found  to  have  a  stronger  positive  effect  on  salesperson  performance  than  negative 
feedback,  which  had  no  effect  (Chakrabarty  et  al.,  2008;  Jaworski  &  Kohli,  1991). 

Moreover,  studies  examining  the  interactive  effect  of  feedback  sign  and  task  type  on  motivation 
and  performance  have  found  support  for  the  use  of  positive  feedback  (Kluger  &  DeNisi,  2005; 
Van-Dijk  &  Kluger,  2004).  In  two  studies,  researchers  found  that  individuals  involved  in  a 
promotion  task  (i.e.,  creative  or  complex  task)  increase  their  motivation  and  subsequent 
performance  with  positive  feedback  rather  than  negative  feedback.  Additionally,  individuals 
who  were  in  a  promotion  focus  (fulfilling  a  desire)  also  increased  their  motivation  with  positive 
feedback  rather  than  negative  feedback  (Kluger  &  DeNisi,  2005;  Van-Dijk  &  Kluger,  2004). 
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Relationships  between  feedback  sign  and  other  variables  that  indirectly  influence  perfonnance 
have  also  been  found  (Ilies  et  al.,  2007;  Karl  et  ah,  1993).  Ilies  et  al.  (2007)  found  that  positive 
feedback  increased  positive  affectivity  while  negative  feedback  increased  negative  affectivity. 
These  affective  reactions  can  be  expected  to  influence  an  individual’s  performance. 

Furthermore,  previous  studies  suggest  that  people  process  positive  and  negative  feedback 
information  differently  (Ilies  et  ah,  2007).  Because  learning  interpersonal  skills  in  a  computer- 
based  adaptive  environment  is  a  creative  and  complex  task  it  may  be  that  positive  feedback  will 
be  more  effective  in  improving  performance.  In  a  highly  interactive  and  challenging 
environment  positive  feedback  will  serve  as  a  signal  of  good  perfonnance  and  in  turn  provide  the 
motivation  necessary  to  continue  working  toward  the  goal.  Therefore,  if  positive  feedback  is 
necessary  to  increase  performance  then  the  following  hypothesis  will  be  supported: 

Hypothesis  3:  Following  the  social  cognitive  theory,  trainees  in  the  positive  feedback 
group  will  have  a  higher  change  in  performance  for  the  pre-  and  post-test  and  Mission  1 
and  Mission  2  scores  than  trainees  in  the  negative  feedback  group. 

Control  theory.  Research  also  suggests  that  negative  feedback  is  necessary  to  enhance 
performance,  contradicting  the  results  discussed  in  the  previous  section.  This  is  explained  by 
control  theory  (Carver  &  Scheier,  1981).  The  main  focus  of  control  theory  is  the  negative 
feedback  loops  made  up  of  four  components:  input  function,  standard,  comparator  and  output 
function.  The  input  function  senses  information  outside  of  the  system  and  brings  it  into  the  loop. 
This  is  synonymous  with  one’s  current  state  or  behavior.  The  standard  is  the  goal  that  the 
individual  has  set  up.  The  function  of  the  comparator  is  to  evaluate  the  input  and  the  standard  to 
determine  whether  there  are  any  discrepancies.  If  there  is  a  discrepancy,  the  output  function  is 
activated  to  bring  input  in  line  with  the  standard  and  in  turn  eliminate  any  discrepancy.  Another 
option  instead  of  changing  the  input  is  to  change  the  standard  or  goals  the  individual  may  have. 

If  the  comparator  does  not  find  a  discrepancy  then  the  system  stays  at  the  same  level. 

Additionally,  a  central  tenet  of  control  theory  is  that  goals  are  hierarchically  arranged.  Short¬ 
term  behavioral  goals  are  regulated  by  feedback  loops  at  the  bottom  of  the  hierarchy  while  long¬ 
term  abstract  goals  are  regulated  by  feedback  loops  at  the  top  of  the  hierarchy.  These  goals  are 
interrelated  such  that  lower  goals  represent  the  means  by  which  higher  level  goals  are  achieved. 
In  general,  this  theory  provides  a  causal  explanation  between  feedback  sign  and  performance. 
Lord  and  Levy  (1994)  proposed  a  hybrid  version  of  control  theory  by  incorporating  control 
theory  principles  with  various  elements  of  human  information  processing  theory  to  explain 
coordination  in  cognition,  motivation,  and  behavior.  They  suggested  a  flexible  and  more  loosely 
constrained  system  as  opposed  to  the  mechanical  responses  of  the  control  system.  In  this  hybrid 
version  other  constraints  from  tasks,  social  environments,  or  a  person’s  physical  or  affective 
systems  are  considered  in  explaining  how  goals  emerge  at  the  hierarchical  levels.  In  line  with 
control  theory,  the  authors  suggested  that  feedback  needs  to  interrupt  processing  in  order  to  shift 
attention  and  improve  perfonnance.  This  intenuption  of  processing  will  more  likely  occur  if  the 
feedback  is  negative. 

Studies  have  shown  support  for  the  contention  that  negative  feedback  increases  performance. 
However,  little  research  has  been  conducted  during  the  learning  of  an  interpersonal  or  complex 
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skill.  Most  of  the  literature  has  been  conducted  within  a  training  design  that  consists  of  simple 
tasks  or  motor  tasks  (Chadda,  1991;  Kluger  &  DeNisi,  2005;  Mesch,  1994;  Wiener,  1963; 

Wiener  &  Attwood,  1968).  The  few  studies  that  involved  the  influence  of  negative  feedback  on 
interpersonal  or  complex  skills  have  focused  on  perfonnance  appraisals  or  upward  feedback 
programs  (Atawater,  1995;  Johnson  &  Ferstl,  1999;  Reilly  et  al.,  1996;  Smither  et  ah,  1995; 
Walker  &  Smither,  1999). 

Studies  that  have  examined  upward  feedback,  which  occurs  when  subordinates  provide  feedback 
to  a  manager  or  supervisor,  refer  to  negative  feedback  as  the  discrepancy  between  the 
supervisor’s  self-ratings  and  the  ratings  given  to  them  by  a  subordinate.  In  other  words,  if  a 
supervisor’s  self-ratings  are  high  but  the  feedback  provided  by  the  subordinates  indicates  low 
ratings  then  this  is  considered  negative  feedback.  In  a  study  by  Johnson  and  Ferstl  (1999), 
employees  rated  their  supervisors  under  the  categories  of  leadership,  people  management  (e.g., 
coaching  evaluation,  counseling),  people  development,  and  communications.  The  study  showed 
that  mangers  who  received  “negative”  upward  feedback  improved  performance  from  one  year  to 
the  next. 

Similar  findings  have  been  found  in  longitudinal  upward  feedback  studies.  Managers  who 
received  lower  performance  ratings  from  subordinates  in  comparison  to  their  self-ratings  (i.e., 
negative  feedback)  had  increased  performance  during  a  2.5  year  program  than  did  managers  who 
received  higher  performance  ratings  (i.e.,  positive  feedback)  (Reilly  et  al.,  1996).  These  results 
were  also  found  in  a  5-year  program  with  annual  administrations  of  upward  feedback.  Managers 
initially  rated  poor  or  moderate  showed  significant  improvements,  as  compared  to  managers 
rated  highly  (Walker  &  Smither,  1999).  Consistent  with  the  previous  studies,  Smither  et  al. 
(1995)  and  Atwater,  Roush,  and  Fischthal  (1995)  found  that  managers  and  student  leaders  with 
initial  low  or  moderate  perfonnance  had  a  greater  improvement  in  their  performance  after 
receiving  negative  feedback  than  did  managers  with  high  performance. 

The  positive  influence  of  negative  feedback  on  performance  has  also  been  found  with  individuals 
working  in  groups.  Mesch,  Farh,  and  Podsakoff  (1994)  and  Podsakoff  and  Farh  (1989) 
examined  the  effect  of  feedback  sign  but  on  group  goal-setting,  strategy,  and  performance. 
Participants  were  provided  with  positive  and  negative  feedback.  Both  studies  found  that 
negative  feedback  groups  perfonned  at  higher  levels,  set  higher  goals,  and  developed  more 
strategies  than  did  the  positive  feedback  groups. 

The  previous  studies  support  the  notion  that  negative  feedback  will  inform  individuals  of  the 
discrepancy  between  their  goals  and  perfonnance  and  in  turn  motivate  individuals  to  improve 
their  performance.  Based  on  the  implications  of  control  theory,  the  following  hypothesis  will  be 
tested: 


Hypothesis  4:  Following  the  control  theory,  trainees  in  the  negative  feedback  group  will 
have  a  higher  change  in  performance  for  the  pre-  and  post-test  and  Mission  1  and  Mission 
2  scores  than  trainees  in  the  positive  feedback  group. 
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Feedback  Orientation 


Researchers  suggest  that  training  effectiveness  may  be  improved  by  focusing  on  the  training 
design  features  and  the  interaction  of  the  design  features  with  individual  differences  (Baldwin  & 
Ford,  1988;  Noe  &  Schmitt,  1986).  This  is  because  training  methods  may  be  differentially 
effective  for  different  individuals.  Furthermore,  the  usefulness  of  feedback  in  general  and  its 
influence  on  performance  has  been  shown  to  depend  upon  both  the  nature  of  the  feedback 
stimulus  and  the  recipient  (Herold,  Parsons,  &  Rensvold,  1996;  Ilgen  et  ah,  1979).  Research  has 
demonstrated  that  considering  personality  factors  is  an  important  part  of  training  because  it  can 
impact  the  ability  to  acquire  skills  (Oakes,  2001).  Oakes  (2001)  found  that  some  personality 
factors  (reasoning  and  apprehension)  positively  correlate  with  skill  acquisition,  and  skill 
acquisition  can  predict  the  level  of  job  performance.  Thus,  interpretation  of  the  feedback  sign 
and  feedback  frequency  may  depend  on  individual  differences  and  this  will  influence  how  the 
feedback  is  used  for  development.  Although  there  have  been  calls  for  this  type  of  research 
(Fedor,  1991)  relatively  few  studies  have  focused  on  how  responses  to  feedback  may  differ 
across  individuals  (Herold  &  Fedor,  1998;  Renn  &  Prien,  1995). 

Furthermore,  the  areas  of  research  that  have  examined  individual  characteristics  have  been 
mixed  on  which  characteristics  are  most  important.  Studies  in  the  feedback  and  training 
literature  have  differed  on  the  individual  characteristics  that  are  focused  on,  which  makes 
generalizability  of  the  findings  difficult. 

For  example,  feedback  researchers  have  looked  at  the  impact  of  age  (Chadda,  1991),  feedback 
source  credibility  and  feedback  quality  (Steelman  &  Rutkowski,  2003),  and  personality 
characteristics  such  as  pessimism  and  optimism  of  the  feedback  receiver  (Szalma,  Hancock, 
Dember,  &  Warm,  2006).  On  the  other  hand,  the  training  literature  has  looked  at  trainee 
characteristics  such  as  ability,  personality,  motivation  (Baldwin  &  Ford,  1998),  locus  of  control, 
conscientiousness,  anxiety,  cognitive  ability,  learning  goal  orientation  (Klein,  2006),  and 
instrumentality  (Tziner,  Fisher,  Senior,  &  Weisberg,  2007),  which  have  been  found  to  influence 
both  motivation  and  performance.  Additionally,  in  a  literature  review  conducted  by  Cheng  and 
Ho  (2001)  the  authors  found  that  the  most  studied  individual  characteristics  in  the  training 
literature  were  locus  of  control,  self-efficacy,  motivation,  career/job  attitudes,  organizational 
commitment,  and  decision/reaction  to  training.  However,  the  issue  with  all  of  these  trainee 
characteristics  is  that  none  of  them  directly  take  into  account  the  role  of  one’s  proclivity  to  value, 
accept,  and  use  feedback.  It  is  necessary  to  study  individual  characteristics  more  closely  related 
to  feedback  to  better  understand  how  individual  differences  may  influence  the  feedback  process 
(Linderbaum  &  Levy,  2007). 

Feedback  orientation  in  general  refers  to  an  individual’s  receptivity  to  feedback  (London  & 
Smither,  2002).  Feedback  orientation  blends  the  typical  training  characteristics  studied  such  as 
ability  (using  the  feedback  received),  valence,  and  instrumentality  (belief  that  the  feedback  offers 
insight  and  is  useful)  but  it  allows  more  focus  on  the  feedback  aspect  of  the  training.  People  who 
have  strong  feedback  orientations  value  feedback;  they  welcome  feedback  naturally  and  they 
seek  out  feedback.  Individuals  with  a  strong  feedback  orientation  process  feedback  carefully  and 
deeply  and  they  want  to  know  what  it  means.  Also,  these  individuals  want  to  know  why  people 
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feel  the  way  they  do  about  their  performance  (London  &  Smither,  2002).  London  and  Smither 
(2002)  described  feedback  orientation  as  a  multi-dimensional  construct.  The  dimensions 
include:  liking  feedback  or  an  overall  positive  affect  toward  feedback,  behavioral  propensity  to 
seek  feedback,  cognitive  propensity  to  process  feedback  mindfully  and  deeply,  sensitivity  to 
others’  view  of  oneself,  perceived  value  of  feedback  or  belief  that  feedback  offers  insight  that 
may  help  them  become  more  effective,  and  feeling  accountable  to  use  the  feedback. 

Linderbaum  and  Levy  (2007)  developed  and  validated  a  feedback  orientation  scale.  They 
examined  the  previous  dimensions  suggested  by  London  and  Smither  (2002),  as  well  as  included 
additional  dimensions  after  further  literature  review.  The  authors  tested  this  model  and 
concluded  that  four  dimensions  make  up  the  Feedback  Orientation  Scale.  These  dimensions  are: 
utility,  accountability,  social  awareness,  and  feedback  self-efficacy.  Utility  refers  to  an 
individual’s  tendency  to  believe  that  feedback  is  instrumental  in  achieving  goals.  Accountability 
is  an  individual’s  tendency  to  feel  a  sense  of  obligation  to  act  on  the  feedback  given  to  them. 
Social  awareness  refers  to  an  individual’s  tendency  to  use  feedback  to  be  aware  of  other’s  views 
of  oneself  and  to  be  sensitive  to  these  views.  Lastly,  feedback  self-efficacy  is  an  individual’s 
tendency  to  have  confidence  in  dealing  with  feedback  situations  and  feedback  itself. 

An  individual  with  high  feedback  orientation  is  similar  to  those  with  high  learning  goal 
orientations  where  learners  focus  on  gaining  competence,  developing  new  skills  and  learning 
from  experience  (Klein,  2006).  Research  has  shown  support  for  this  relationship.  Linderbaum 
and  Levy  (2007)  found  a  positive  relationship  between  feedback  orientation  and  learning  goal 
orientation.  They  also  found  that  feedback  orientation  was  positively  correlated  to  job 
involvement,  protestant  work  ethic,  general  self-efficacy,  and  positive  affect.  These  results 
further  support  the  notion  that  personality  characteristics  closely  related  to  feedback  may  be 
related  to  motivation  and  performance. 

Although  few  studies  have  been  conducted  examining  the  role  of  feedback  orientation, 
researchers  suggest  that  improvement  in  performance  is  more  likely  for  some  feedback  recipients 
than  others  (Smither  et  ah,  2005).  Perfonnance  improvements  are  more  likely  to  occur  when 
participants  react  positively  towards  feedback,  when  they  take  actions  that  lead  to  skill 
improvement,  and  when  they  have  a  positive  feedback  orientation. 

Moreover,  researchers  suggest  a  relationship  exists  among  feedback  orientation,  feedback 
frequency,  and  feedback  sign  that  warrants  the  study  of  feedback  orientation  within  a  training 
context.  London  and  Smither  (2002)  wrote  that  feedback  is  not  only  a  one-time  event;  but 
instead  it  is  a  frequent  intervention.  The  effects  of  feedback  occur  over  time  as  the  individual 
receives,  absorbs,  and  uses  the  feedback.  The  way  the  individual  responds  to  frequent  feedback 
is  further  influenced  by  an  individual’s  feedback  orientation,  which  may  play  a  role  in  the 
individual’s  anticipation  of  feedback,  interpretation  of  feedback,  and  goal  setting  based  on 
feedback.  Thus,  the  researchers  support  the  notion  that  the  frequency  of  feedback  (infrequent 
feedback  versus  frequent  feedback)  may  motivate  trainees  differently  depending  on  their 
feedback  orientation.  That  is,  those  who  value  feedback  (have  a  high  feedback  orientation)  may 
be  less  motivated  when  receiving  infrequent  feedback  than  when  receiving  frequent  feedback 
during  training.  On  the  other  hand,  those  who  do  not  value  feedback  (have  low  feedback 


20 


orientation)  may  be  less  motivated  when  receiving  frequent  feedback  rather  than  infrequent 
feedback. 

Additionally,  London  and  Smither  (2002)  suggested  that  feedback  orientation  may  influence  the 
effect  of  feedback  sign  on  performance.  Individuals  are  not  always  receptive  to  feedback,  in 
other  words,  they  may  have  low  feedback  orientation.  Individuals  with  low  feedback  orientation 
may  require  more  negative  feedback  to  find  the  feedback  meaningful  enough  to  alter  their 
performance.  This  may  be  especially  true  if  the  feedback  is  referring  to  a  personality  trait. 
Therefore,  the  authors  indicate  that  reactions  to  negative  feedback  would  be  related  to  the 
strength  and  consistency  of  the  feedback  and  the  individual’s  feedback  orientation.  Whereas 
people  with  high  feedback  orientation  may  value  both  negative  and  positive  feedback,  those  with 
low  feedback  orientation  may  be  less  motivated  by  negative  feedback  rather  than  positive 
feedback. 

Hypothesis  5:  Feedback  orientation  will  moderate  the  relationship  between  training 
design  (feedback  sign  and  feedback  frequency)  and  performance. 

Hypothesis  5(a):  Individuals  with  high  feedback  orientation  in  the  frequent  feedback 
condition  will  have  a  higher  change  in  performance  for  the  pre-  and  post-test  and  Mission 
1  and  Mission  2  scores  than  individuals  with  high  feedback  orientation  in  the  infrequent 
feedback  condition.  Individuals  with  low  feedback  orientation  in  the  infrequent  feedback 
orientation  will  have  higher  change  in  performance  for  the  pre-  and  post-test  and  Mission 
1  and  Mission  2  scores  than  individuals  with  low  feedback  orientation  in  the  frequent 
feedback  condition. 

Hypothesis  5(b):  Individuals  with  high  feedback  orientation  in  the  positive  condition 
will  perform  at  the  same  level  as  high  feedback  orientation  individuals  in  the  negative 
feedback  condition.  Individuals  with  low  feedback  orientation  in  the  positive  feedback 
condition  will  have  a  higher  change  in  performance  for  the  pre-  and  post-test  and  Mission 
1  and  Mission  2  scores  than  individuals  with  low  feedback  orientation  in  the  negative 
feedback  condition. 

London  and  Smither  (2002)  proposed  that  an  interaction  may  occur  between  feedback  sign  and 
feedback  frequency.  They  suggested  that  individuals  with  low  feedback  orientation  may  require 
a  higher  frequency  of  negative  feedback  to  improve  their  performance  in  comparison  to  those 
with  high  feedback  orientation.  Furthermore,  previous  studies  have  shown  that  feedback  sign 
may  interact  with  other  factors  such  as  task  type  (Kluger  &  Van-Dijk,  2005;  Van-Dijk  &  Kluger, 
2004).  For  this  reason,  as  an  exploratory  approach  this  research  will  examine  the  interaction 
between  feedback  sign  and  frequency. 


Methodology 

The  training  program  used  for  the  current  research  was  the  Virtual  Environment  Composable 
Training  for  Operational  Readiness  (VECTOR)  developed  for  the  U.S.  Anny  Research  Institute 
for  the  Behavioral  and  Social  Sciences  by  Chi  Systems,  Inc.  VECTOR  is  designed  to  provide 
game -based  adaptive  training  in  interpersonal  skills  through  the  application  of  experiential, 
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scenario-based  training  in  a  virtual  environment.  This  platform  was  chosen  because  it  serves  as  a 
generic  interpersonal  skills  training  environment.  It  provides  content  management  tools  that 
allow  editing  and  authoring  of  scenario  interactions.  More  specifically,  VECTOR  provides 
direct  control  of  the  behavior,  dialog,  emotional  state,  and  predispositions  of  the  characters 
within  the  scenario,  which  allows  the  trainer  to  design  training  scenarios  around  specific  learning 
objectives.  The  authoring  capabilities  also  include  when  and  where  the  characters  will  appear  in 
the  game  and  the  feedback  that  can  be  given  to  the  trainee.  Overall,  due  to  its  authoring 
capabilities  VECTOR  was  found  to  be  an  ideal  platform  for  this  research  application. 

The  simulation  used  in  VECTOR  is  based  on  real  world  events  and  input  from  military  experts. 
The  purpose  of  the  simulation  was  to  improve  an  individual’s  interpersonal  skills,  which  include 
communication  and  negotiation.  The  simulation  reinforced  essential  skills  when  negotiating  or 
interacting  with  other  individuals,  such  as  developing  relationships  over  time,  building  and 
gaining  the  partner’s  trust,  and  planning  what  to  accomplish,  when  to  escalate,  and  when  to  walk 
away.  More  specifically,  the  training  objectives  in  the  videogame  were  based  on  a  hierarchical 
tree-structure  (See  Figure  1  below). 

The  highest-level  of  abstraction  were  the  Terminal  Training  Objectives,  which  represent  the  core 
aspects  of  the  knowledge,  skills,  and  abilities  being  trained.  These  were:  (1)  understanding  the 
proper  order  of  a  negotiation  event  or  conversation,  (2)  knowing  proper  communication 
etiquette,  and  (3)  ability  to  build  cooperation.  The  lower-level  decompositions  of  the  Terminal 
Training  Objectives  are  the  Subordinate  Training  Objectives,  which  are  linked  to  the  terminal 
objectives.  These  were:  (1)  understanding  the  proper  order  of  a  negotiation  event  or 
conversation,  (2)  knowing  proper  communication  etiquette,  and  (3)  ability  to  build  cooperation. 
The  lower-level  decompositions  of  the  Terminal  Training  Objectives  are  the  Subordinate 
Training  Objectives,  which  are  linked  to  the  terminal  objectives.  These  were:  (1)  displaying 
appropriate  greeting  and  small  talk,  (2)  knowing  how  to  follow  the  lead  in  a  conversation,  (3) 
being  able  to  acknowledge  someone’s  perspective  in  a  conversation,  (4)  showing  patience  in  a 
conversation,  (5)  developing  a  “Win/Win”  negotiation  strategy,  and  (6)  serving  as  a  peacemaker. 
Table  A1  offers  a  detailed  explanation  of  each  of  the  Subordinate  Training  Objectives.  The 
lowest-level  of  the  hierarchy  is  the  Enabling  Objectives.  These  represent  all  the  objectives  given 
to  the  player  in  the  game  that  must  be  satisfied  in  order  for  the  linked  sub-ordinate  training 
objective  to  be  satisfied.  An  example  of  an  objective  is  “To  get  participation  in  getting  medical 
supplies.”  VECTOR  tracks  player  performance  through  the  accomplishment  of  these  objectives. 
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Interpersonal  Skills  Training 


Terminal  Objectives: 


1.  Proper  Order  of  Negotiation 

or  Conversation  Events 
— 

2.  Proper  Communication 
Etiquette 

3.  Building  Cooperation 

Sub-ordinate  Obiectives: 

1 

1.1.  Greeting/Small  Talk 

2.1.  Acknowledge  Someone's 
Perspective 

3.1.  "Win/Win"  Negotiation 
Strategy 

1.2.  Following  the  Lead 

2.2.  Patience  in  Conversation 

3.2.  Serve  as  Peacemaker 

i  I 


Enabling  Objectives: 


Mission  1:  Establish  Medico I  Screening  Site 


1.3.  Learn  why  there  is  no  clinic 


2.3.  Determine  who  in  the 
community  can  help  In 
establishing  site 


Mission  2:  Resolve  Safety  Concerns  at  Medical  Screening  Site 


14.  Learn  what  the  safety 
concerns  are.  Why  is  the 
site  not  being  used? 


2.4.  Determine  how  to  best 
resolve  safety  concerns. 
Who  is  best  to  talk  to  and 
what  can  be  done? 


3.3.  Get  cooperation  in  exchange 
for  looking  into  supply  issues 


3.4.  Get  participation  in  getting 
medical  supplies 


3.S.  Calm  the  doctors  so  they 
stop  their  threats  and  return  to 
work 


3.6.  Get  cooperation  and 
increased  protection  from 
police  forces 


Figure  1.  Overview  of  training  objectives 


Overview  of  Structure  of  the  Game 

The  videogame  for  this  research  consists  of  two  missions.  Both  missions  were  the  same  length 
and  trained  the  same  objectives  (See  Table  1).  The  only  difference  between  the  two  missions  is 
the  Enabling  Objectives  that  were  given  to  the  trainee.  The  game  began  with  a  Mission 
Statement  that  included  the  details  of  the  goals  and  background  infonnation.  The  first  mission 
was  to  establish  a  medical  screening  site  in  the  village  of  Kahzar  to  establish  rapport  and  build 
confidence  with  the  local  populace.  Trainees  were  told  that  the  medical  screening  station  would 
provide  basic  medical  screening  and  first  aid  to  local  women,  infants,  and  children.  To  succeed 
in  the  mission  participants  were  told  they  must  observe  local  customs  and  courtesies. 
Additionally,  participants  were  required  to  meet  and  establish  rapport  with  the  local  populace.  In 
the  second  mission  the  medical  screening  site  has  been  set  up  but  they  are  told  there  are  security 
issues.  The  doctors  and  nurses  are  scared  to  go  to  the  clinic  and  it  is  rarely  ever  open.  The 
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trainee  must  resolve  the  safety  concerns  at  the  medical  screening  site  and  get  it  operating  again. 
Once  the  mission  objective  was  reviewed  the  next  step  was  the  tutorial  screen.  The  tutorial 
screen  introduced  the  trainee  to  interpersonal  skills  training.  It  reviewed  important  elements  of 
the  interpersonal  skills  training  (i.e.,  terminal  objectives)  and  went  into  detail  about  the  sub¬ 
ordinate  objectives  that  the  trainee  was  expected  to  accomplish  (Table  1).  The  tutorial  screen 
also  explained  the  meaning  of  the  performance  measures  that  were  provided  at  the  end  of  the 
game.  Both  missions  included  the  same  tutorial  screen.  After  reviewing  the  mission  statement 
and  the  tutorial  screen  the  trainee  began  the  simulation. 

Each  mission  consisted  of  six  interactions  between  the  trainee  and  a  town  resident.  The  trainee 
had  to  move  the  avatar  through  the  environment  to  find  a  town  resident  to  speak  with.  During 
each  interaction,  the  trainee  was  expected  to  accomplish  the  Enabling  Objectives  by  successfully 
using  the  appropriate  interpersonal  skills  (sub-ordinate  objectives).  The  response  of  the  town 
resident  adapted  to  the  behavior  and  choices  the  trainee  made.  For  example,  the  first  interaction 
with  a  town  resident  in  Mission  1  required  the  trainee  to  find  out  why  there  was  no  medical 
screening  site.  However,  the  trainee  had  to  accomplish  this  while  demonstrating  the  appropriate 
response  to  greetings  and  small  talk.  When  the  trainee  approached  the  town  resident  they  were 
greeted  with  “Hello.”  Then,  the  trainee  was  given  four  response  options:  (1)  greet  the  character 
using  a  customary  local  greeting  by  clicking  on  the  “[Greet  in  Arabic]”,  (2)  begin  the 
conversation  by  saying,  “Hello,  do  you  know  why  a  clinic  has  not  been  established  in  this 
village?”  (3)  use  the  local  greeting  “[Greet  in  Arabic]”  in  addition  to  saying,  “I  would  like  to  fix 
the  problem  with  the  clinic,  but  I  would  like  your  cooperation  to  do  so,”  or  (4)  “Hello.  I  hope  you 
are  doing  well  today.”  If  the  player  chose  an  appropriate  response  such  as  response  number  1 : 
“[Greet  in  Arabic]”,  then  the  town  resident  responded  with,  “This  is  good  yes.  Thank  you  for  this 
sign  of  respect.”  On  the  other  hand,  if  the  trainee  chose  an  incorrect  option  such  as  response 
number  2:  “Hello.  Do  you  know  why  a  clinic  has  not  been  established  in  this  village?”  then  the 
town  resident  responded  with  “Let’s  take  the  time  and  just  be  social.  Please  it’s  a  hot  day 
outside  and  we  haven’t  taken  the  time  to  know  each  other”.  Overall,  if  the  player  continued  to 
display  the  appropriate  greetings  and  small  talk  then  the  town  resident  remained  happy  and 
directed  the  trainee  towards  another  town  resident  who  would  know  why  no  clinic  had  been 
established.  However,  if  the  trainee  did  not  meet  the  objectives  and  upset  the  town  resident  then 
this  had  a  negative  influence  on  the  interaction  with  the  next  town  resident.  The  second 
interaction  was  with  a  local  doctor.  The  greeting  from  the  doctor  depended  on  the  results  of  the 
first  interaction.  For  example,  if  the  first  interaction  ended  positively  by  accomplishing  the 
objective  then  the  doctor  would  greet  the  trainee  with,  “Hello,  Galeb  tells  me  good  things  about 
you.”  Thus,  the  characters  in  the  game  respond  in  relation  to  the  interpersonal  skills  that  the 
trainee  exhibited.  Furthermore,  feedback  could  be  provided  at  the  end  of  each  interaction.  The 
feedback  was  positive  or  negative  depending  on  whether  the  objective  was  accomplished.  The 
feedback  was  provided  in  the  form  of  a  voice-over  that  served  as  a  coach.  More  details  on  the 
feedback  intervention  will  be  provided  in  subsequent  sections  of  this  document.  At  the  end  of 
the  mission  or  the  sixth  interaction,  a  summary  screen  was  provided  to  the  trainee  titled  Training 
Objective  Results.  This  screen  lists  the  sub-ordinate  training  objectives  and  for  each  objective 
whether  the  trainee  succeeded  or  failed.  It  also  included  the  feedback  that  was  provided  as  a 
voice-over  after  every  interaction. 
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Table  1.  Detailed  Description  of  Sub-Ordinate  Objectives 


Sub-Ordinnte  Objective 

Description 

Responding  to  Greetings/Small  Talk 

Demonstrate  the  appropriate  response  to  greetings  and  small  talk  during  a 
fonnal  or  informal  situatioa 

Following  the  Lead  in  Conversation 

In  a  new  setting,  take  the  lead  from  the  other  person  in  a  social  conversation. 
What  he/slie  brings  up  should  be  the  topic  of  discussion  after  that.  Refrain  from 
switching  the  topic. 

Acknowledge  Someone's  Perspective 

Identify’  when  you  are  not  in  agreement  with  the  other  person's  statement  or 
perspective  without  insulting  or  upsetting  the  person. 

Patience  in  Conversation 

Demonstrate  patience  in  conversation  by  refraining  front  getting  upset  or 
irritated  during  a  heated  conversation.  Stay  focused  on  the  topic  of 
conversation. 

"Win-Win"  Negotiation  Strategy 

Use  a  “win-win"  negotiation  strategy  where  the  interests  and  positions  of  both 
parties  to  the  negotiation  closely  align.  This  leads  to  accomplishing  multiple 
issues  and  building  relationships. 

Serve  as  Peacemaker 

Demonstrate  the  ability  to  reduce  tension  and  conflict  between  people  during  a 
cooperative  effort. 
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Research  Design 


Participants.  Participants  were  recruited  by  posting  flyers  around  campus  at  two 
universities  in  Florida.  Recruitment  also  included  posting  infonnation  about  the  experiment  on  a 
university  online  forum  and  on  www.craigslist.org.  Participants  were  compensated  $10  per  hour 
for  their  participation.  A  total  of  159  participants  were  included  in  this  research.  The  ages 
ranged  from  18-31  with  a  mean  of  22.  The  sample  was  62%  male  and  54%  Caucasian.  In  the 
sample,  98%  of  the  participants  stated  they  did  not  have  previous  experience  with  interpersonal 
skills  training  and  20%  stated  they  never  played  computer-based  videogames. 

Design.  The  design  of  the  research  was  based  on  the  voice-over  feedback  that  was 
provided  to  the  trainee  after  each  interaction  with  a  town  resident.  The  design  was  a  2  x  2 
mixed-design  with  the  dependent  variable  as  a  performance  change  score.  Participants  were 
assigned  to  one  of  four  conditions:  (1)  frequent  positive  feedback,  (2)  infrequent  positive 
feedback,  (3)  frequent  negative  feedback,  or  (4)  infrequent  negative  feedback.  Participants  were 
randomly  assigned  to  the  conditions  using  a  research  randomizer  tool  from  Research  Randomizer 
(Urbaniak  &  Pious,  1997-2010)  website.  The  tool  generates  randomized  number  sets  using  the 
“math.random”  method  within  the  JavaScript  programming  language. 

All  feedback  provided  was  veridical,  that  is  feedback  was  based  on  the  trainee’s  actual 
performance.  Those  in  the  positive  feedback  conditions  received  favorable  feedback  when  they 
performed  well;  they  did  not  receive  negative  feedback  when  they  performed  poorly.  Those  in 
the  negative  feedback  conditions  received  unfavorable  feedback  when  they  perfonned  poorly  but 
did  not  receive  positive  feedback  after  performing  well.  Those  in  the  frequent  feedback 
condition  received  feedback  after  every  interaction,  as  well  as  the  summary  screen  feedback  if 
their  performance  corresponded  to  the  positive/negative  condition  to  which  they  were  initially 
randomly  assigned.  Additionally,  participants  in  the  infrequent  feedback  condition  only  received 
feedback  as  the  summary  screen  for  their  perfonnance  that  corresponded  to  the  positive/negative 
condition  to  which  they  were  initially  randomly  assigned.  For  example,  participants  in  the 
infrequent  feedback  condition  who  failed  the  objectives  would  only  receive  feedback  if  they 
were  assigned  to  the  negative  feedback  condition.  Similarly,  if  they  performed  well  they  would 
only  receive  positive  feedback  if  they  were  assigned  to  the  positive  feedback  condition. 
Otherwise,  they  would  not  receive  any  feedback.  For  the  frequent  feedback  condition, 
participants  who  received  less  than  3  of  the  12  available  feedback  interventions  were  removed 
from  subsequent  data  analysis.  There  were  a  total  of  14  feedback  interventions  with  2  of  the 
interventions  given  automatically  regardless  of  the  participants’  performance  in  the  form  of  the 
summary  screen. 

An  example  of  positive  feedback  based  on  the  “Win-Win  Negotiation  Strategy”  objective  is, 
“Good  work.  You  performed  better  than  average.  You  exhibited  a  “win-win”  negotiation 
strategy.  You  considered  the  interests  of  both  parties  and  worked  towards  finding  a  solution.”  It 
is  important  to  note  that  negative  feedback  may  have  an  effect  on  performance.  Furthermore, 
depending  on  its  frame  it  may  also  have  a  negative  affect.  However,  in  order  to  provide  sufficient 
information  for  the  individual  to  determine  how  they  were  performing,  negative  content  was 
used  to  operationalize  negative  feedback.  Additionally,  a  nonnative  portion  was  included  in  the 
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feedback  to  ensure  that  individuals  recognize  they  are  deviating  from  the  standard.  An  example 
of  negative  feedback  for  the  same  objective  is,  “Your  performance  was  poor.  You  performed 
below  average.  You  did  not  consider  the  other  person’s  position.  As  a  result,  you  did  not  solve 
the  problem  or  build  a  relationship  with  the  local.  Go  speak  to  Ameen.”  Participants  who 
received  negative  feedback  were  instructed  on  whom  to  speak  to  next.  This  is  not  included  in  the 
positive  feedback  because  if  the  appropriate  interpersonal  skills  were  used,  then  the  town 
resident  would  notify  the  trainee  where  to  go  next.  The  perfonnance  feedback  for  each  objective 
remained  the  same  for  both  missions. 

Procedure.  Prior  to  the  investigation,  a  pilot  test  was  conducted  to  test  the  procedures 
and  methods  of  the  focal  investigation.  More  specifically,  the  pilot  test  assessed  the 
programming  for  all  the  interactions  to  ensure  the  videogame  ran  appropriately  and  was 
understood  by  the  participants.  It  also  assessed  the  effectiveness  of  the  manipulations  as  well  as 
the  usefulness  of  the  measures  chosen.  Any  issues  with  the  program,  procedures,  or  methods 
were  corrected  prior  to  conducting  the  focal  investigation. 

The  focal  investigation  was  conducted  at  the  U.S.  Army  Research  Institute  research  laboratories 
in  Orlando,  Florida.  The  room  used  contained  three  long  tables  with  six  computers  placed 
against  the  walls.  Each  computer  station  was  equipped  with  the  videogame  VECTOR, 
headphones  and  an  instruction  packet  on  how  to  use  VECTOR. 

First,  participants  were  given  informed  consent  forms  that  included  a  description  of  the  research. 
Following  this,  participants  were  given  the  self-efficacy  scale,  which  took  5  minutes  to  complete. 
Then,  they  were  given  the  VECTOR  scenario  questionnaire  pretest,  which  took  15  minutes  to 
complete.  This  test  assessed  their  knowledge  of  interpersonal  skills  prior  to  training.  After 
completion  of  the  test,  participants  were  given  a  hand-out  that  reviewed  what  keyboard  keys 
could  be  used  to  move  around  the  virtual  environment.  They  were  allowed  to  keep  this  hand-out 
for  reference  during  the  game.  Once  trainees  were  familiar  with  the  keys  the  simulation  began. 
Trainees  went  through  the  mission  statement  screen,  tutorial  screen,  and  completed  Mission  1 
and  Mission  2  consecutively.  At  the  completion  of  the  game,  trainees  were  given  the  Feedback 
Orientation  Scale,  which  took  15  minutes  to  complete.  Afterwards,  participants  completed  the 
manipulation  check  scale  and  presence  scale,  which  took  less  than  5  minutes  to  complete.  The 
last  part  of  the  experiment  was  the  posttest  which  took  15  minutes  to  complete.  The  entire 
experiment  took  between  1  and  2  hours. 

Measures. 

Pretest  and  Posttest  VECTOR  Scenario  Questionnaire.  The  pretest  and  posttest  were 
made  specifically  to  measure  the  learning  objectives  of  VECTOR  and  was  used  as  a  measure  of 
training  performance.  It  is  a  scenario-based  questionnaire.  This  test  was  developed  specifically 
for  this  research  thus  no  reliability  or  validity  data  is  available.  However,  the  development  of  this 
scale  involved  the  input  of  subject  matter  experts  (SME)  to  detennine  the  content  validity  of  the 
scenarios.  The  SME’s  had  previously  been  deployed  and  had  similar  incidents  occur  during  their 
deployment.  Participants  read  the  scenarios  that  were  developed  based  off  of  the  objectives  of 
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the  game  and  determined  which  response  option  described  the  most  appropriate  action  for  the 
situation.  There  were  eight  items,  which  were  identical  for  the  pretest  and  posttest. 

VECTOR  Performance  Measures.  Each  interaction  in  VECTOR  was  aligned  with  a 
training  objective  and  was  scored  as  a  dichotomous  pass/fail  for  that  objective.  A  final 
performance  score  for  each  participant  was  calculated  as  the  difference  between  Mission  2  and 
Mission  1  scores  summed  across  all  interactions. 

Feedback  Orientation  Scale.  Feedback  orientation  was  measured  using  the  24  item 
multi-dimensional  Feedback  Orientation  Scale  from  Linderbaum  and  Levy  (2007).  The  items 
were  scored  on  a  6-point  Likert-type  scale  ranging  from  1  {strongly  disagree)  to  6  {strongly 
agree).  A  higher  score  indicated  a  stronger  feedback  orientation.  The  dimensions  of  the  scale 
are  utility,  accountability,  social  awareness  and  feedback  self-efficacy.  A  sample  utility  item  is: 
“Feedback  is  critical  for  improving  performance.”  An  item  from  the  accountability  dimension  is: 
“I  feel  obligated  to  make  changes  based  on  feedback.”  A  sample  item  from  the  social  awareness 
dimension  is:  “Feedback  helps  me  manage  the  impression  I  make  on  others.”  A  feedback  self- 
efficacy  item  is:  “I  believe  that  I  have  the  ability  to  deal  with  feedback  effectively.”  The  overall 
Cronbach  alpha  found  by  Linderbaum  and  Levy  (2007)  was  .86.  The  Cronbach  alpha  for  this 
experiment  was  .89. 

Manipulation  Check.  The  manipulation  check  was  developed  to  assess  the  effectiveness 
of  the  experimental  design  components.  The  survey  consisted  of  14  questions  on  a  6-point 
Likert-type  scale  ranging  from  1  {strongly  disagree)  to  6  {strongly  agree).  A  sample  feedback 
sign  question  is,  “The  information  given  to  me  about  my  performance  showed  that  I  did  well 
interacting  with  the  characters.”  A  sample  feedback  frequency  question  is,  “I  was  not  provided 
with  infonnation  about  my  performance  during  the  game.”  The  manipulation  check  includes  an 
additional  sheet  with  demographic  questions.  The  Cronbach  alpha  for  each  set  of  items  was:  .89 
for  the  positive  feedback  items,  .87  for  the  negative  feedback  items,  .76  for  the  frequent  feedback 
items,  and  .82  for  the  infrequent  feedback  items. 

Presence  Questionnaire.  A  presence  scale  was  used  to  assess  involvement  in  the 
videogame.  This  was  measured  using  four  questions  on  a  7-point  Likert-type  scale  from  Slater, 
Usoh,  and  Steed  (1994).  A  sample  question  is,  “Please  rate  your  sense  of  being  in  the  computer¬ 
generated  world,  where  7  represents  your  nonnal  experience  of  being  in  a  place.”  This  survey 
was  adapted  for  this  research,  thus  no  previous  reliability  or  validity  data  were  available. 
However,  the  only  changes  made  from  the  original  survey  were  changing  the  task  type  to 
“computer-generated  world.”  The  Cronbach  alpha  for  this  research  was  .77. 

Self-efficacy  scale.  Self-efficacy  was  measured  using  a  scale  developed  specifically  for 
the  research,  thus  no  previous  reliability  or  validity  data  were  available.  However,  the  scale  was 
developed  following  the  guidelines  for  developing  self-efficacy  scales  set  by  Bandura  (2006). 

The  scale  consisted  of  10  statements  of  tactics  used  when  communicating  with  someone. 
Participants  were  asked  to  rate  their  confidence  in  successfully  using  each  tactic.  The  scale 
ranged  from  0  {cannot  do  at  all)  to  100  {highly  certain  can  do).  A  sample  item  is,  “Look  for  an 
agreement  that  maximizes  both  of  our  interests.”  The  Cronbach  alpha  for  this  research  was  .84. 
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On-  and  Off-task  attention  scale.  An  on-  and  off-task  attention  scale  was  developed  to 
assess  where  participants  directed  their  attention  during  the  task.  The  measure  was  from  Kanfer, 
Ackerman,  Murtha,  Dugdale,  and  Nelson  (1994)  and  it  was  shortened  and  adapted  for  this 
research.  The  scale  consisted  of  16  questions  on  a  Likert-type  scale  ranging  from  1  {never)  to  8 
{constantly).  A  sample  on-task  attention  item  is,  “I  paid  close  attention  to  the  kind  of  errors  I 
was  making.”  A  sample  off-task  attention  item  is,  “I  lost  interest  in  the  task  for  short  periods.” 
The  scale  was  adapted  to  fit  this  research  by  removing  questions  that  were  specific  to  previous 
studies.  Additionally,  the  task  was  changed  from  ‘flying  planes’  to  ‘accomplishing  the  task’.  In 
the  Kanfer  et  al.  (1994)  study  the  Cronbach  alpha  for  on-task  attention  items  was  .81  and  for  off- 
task  attention  items  was  .59.  The  Cronbach  alpha  for  the  entire  scale  for  this  research  was  .70. 

Results 

Preliminary  Results 

Descriptive  statistics.  Descriptive  statistics  such  as  mean,  standard  deviation,  skewness, 
kurtosis  and  confidence  interval  are  presented  in  Table  2  for  all  of  the  variables  for  the  entire 
sample  and  in  Table  3  for  the  variables  by  feedback  condition.  Additionally,  all  the  scales 
(feedback  orientation  scale,  presence  questionnaire,  self-efficacy  scale  and  attention  scale) 
reached  or  exceeded  a  reliability  of  .60  thus  they  were  considered  acceptable  (Nunally,  1978). 
Table  4  shows  the  correlations  for  all  the  variables  in  this  research. 

Prior  to  conducting  hypothesis  testing,  decision  rules  were  set  to  detennine  the  criteria  for 
removing  data.  Participants  assigned  to  the  frequent  feedback  conditions  were  removed  if  they 
received  less  than  three  feedback  interventions.  Participants  assigned  to  the  infrequent  feedback 
conditions  were  removed  if  they  did  not  receive  any  feedback.  Cases  were  also  removed  if  more 
than  10%  of  the  survey  and  mission  score  data  were  missing.  Any  remaining  missing  data  were 
replaced  using  series  mean.  Following  these  decision  rules,  26  participants  were  removed 
leaving  a  final  sample  size  of  159  with  36  in  the  frequent  positive  condition,  48  in  the  frequent 
negative  condition,  38  in  the  infrequent  positive  condition  and  37  in  the  infrequent  negative 
condition. 

Testing  assumptions.  Assumptions  for  MANOVA  and  regression  analysis  were  tested 
to  determine  if  these  statistical  analyses  were  appropriate.  The  MANOVA  assumptions  that 
were  tested  were  independence  of  observations,  random  sampling  of  data,  multivariate  normality 
and  homogeneity  of  covariance  matrices.  There  was  no  violation  of  independence  of  observation 
or  random  sampling.  Normality  was  determined  by  examining  skewness  and  kurtosis,  histogram 
plots,  normal  Q-Q  plots  and  detrended  normal  Q-Q  plot.  The  plots,  skewness  and  kurtosis  scores 
indicate  that  the  distribution  slightly  deviates  from  normality  for  the  pre-test  scores,  post-test 
scores  and  off  task  attention  scale  scores.  However,  it  was  not  expected  that  this  would  influence 
the  results  due  to  the  large  sample  size  and  MANOVA’s  robustness  against  violations  of  this 
assumption  (Tabachnick  &  Fidell,  2001).  Homogeneity  of  covariance  matrices  was  tested  using 
Box’s  M  test.  This  statistics  tests  the  null  hypothesis  that  the  variance-covariance  matrices  are 
homogeneous.  If  Box’s  M  is  non-significant  then  the  null  hypothesis  cannot  be  rejected.  This 
assumption  was  upheld  (Box’s  M  =  12.44,/?  =  .20). 
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The  assumptions  tested  to  detennine  if  a  regression  analysis  was  appropriate  were  linear 
relationship  between  the  independent  variable  and  the  dependent  variable,  homogeneity  of 
variance  and  covariance,  nonnal  distribution  of  measures,  uncorrelated  residuals,  normality  of 
the  residuals,  and  that  the  independent  variables  were  not  substantially  correlated.  These 
assumptions  were  tested  by  examining  the  plot  of  the  standardized  residuals  by  standardized 
predicted  value.  The  correlations  of  the  independent  variables  were  also  evaluated.  The  plots 
for  the  change  scores  of  the  pre  and  post  test  showed  slight  heteroscedasticity.  According  to 
Berry  and  Feldman  (1985)  and  Tabachnick  and  Fidell  (2007)  slight  heteroscedasticity  has  little 
effect  on  significance  tests.  For  this  reason,  the  moderated  regression  analysis  was  conducted. 

Manipulation  check.  The  purpose  of  the  manipulation  check  was  to  detennine  whether 
participants  perceived  being  in  the  correct  feedback  intervention  group.  Composite  scores  were 
developed  using  the  items  in  the  manipulation  check  questionnaire  that  represented  each  of  the 
feedback  interventions  (frequent  feedback,  infrequent  feedback,  positive  feedback  and  negative 
feedback).  The  composite  scores  were  compared  to  the  feedback  group  the  individuals  were 
assigned  to  using  independent  sample  t- tests.  Overall,  participants  perceived  themselves  being  in 
the  correct  feedback  intervention  group,  providing  support  for  the  fidelity  of  the  manipulation. 
Individuals  in  the  frequent  feedback  condition  scored  themselves  higher  on  receiving  frequent 
feedback  (M  =  4.26)  than  on  receiving  infrequent  feedback  (M  =  2.35,  t  =  1 1.44,  p  <  .00). 
Individuals  in  the  infrequent  feedback  condition  scored  themselves  higher  on  receiving 
infrequent  feedback  (M  =  4.72)  than  frequent  feedback  (M  =  2.45,  t  =-12.66,  p  <  .00). 

Participants  in  the  positive  feedback  condition  scored  themselves  higher  on  receiving  positive 
feedback  (M  =  4.98)  than  negative  feedback  (M  =  2.69,  t  =  1 1.89,  p  <  .00).  Likewise, 
participants  in  the  negative  feedback  group  scored  themselves  higher  on  receiving  negative 
feedback  (M  =  4.15)  than  positive  feedback  (M  =  1.86,  t  =  -13.13,  p  <  .00). 
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Table  2.  Descriptive  Statistics 


Measures 

M 

SD 

95%  Cl 

Skewness 

Kurtosis 

Positive 

3.76 

1.66 

[3.50,4.02] 

-0.21 

-1.27 

Negative 

3.08 

1.59 

[2.83,3.33] 

0.15 

-1.32 

Frequent 

3.35 

1.42 

[3.13,3.58] 

0.06 

-1.04 

Infrequent 

3.52 

1.59 

[3.27,  3.77] 

0.04 

-1.29 

Self-efficacy 

80.60 

10.55 

[78.94, 

82.25] 

-0.73 

1.23 

FO 

4.99 

0.57 

[4.90,  5.08] 

-0.92 

1.41 

Presence 

4.21 

1.31 

[4.01,4.42] 

-0.52 

0.07 

Pretest 

26.22 

3.49 

[25.67, 

26.77] 

-0.36 

3.39 

Posttest 

28.87 

1.94 

[28.56, 

29.17] 

-1.19 

1.88 

Final  test 

2.65 

3.37 

[2.12,3.17] 

-1.19 

1.88 

Mission  1 

9.96 

1.19 

[9.77,  10.14] 

-0.03 

-0.58 

Mission  2 

10.16 

1.01 

[9.99,  10.31] 

-0.36 

-0.04 

Final  mission 

0.20 

1.44 

[-0.03,  0.43] 

-0.09 

0.12 

On  task 

5.83 

1.01 

[5.67,  5.99] 

-0.31 

-0.35 

Off  task 

2.90 

1.27 

[2.70,3.101 

0.79 

0.89 

Note.  Positive  =  Manipulation  check  scores  for  positive  feedback  condition;  Negative  = 
Manipulation  check  scores  for  negative  feedback  condition;  Frequent  =  Manipulation  check  scores 
for  frequent  feedback  condition;  Infrequent  =  Manipulation  check  scores  for  infrequent  feedback 
condition;  Self-efficacy  =  Self-efficacy  scale;  FO  =  Feedback  orientation  scale;  Presence  = 
Presence  scale;  Pretest  =  Pretest  scores  on  VECTOR  questionnaire;  Posttest  =  Posttest  scores  on 
VECTOR  questionnaire;  Final  test  =  Change  scores  for  pretest  and  posttest;  Mission  1  =  Mission 
1  scores  from  VECTOR;  Mission  2  =  Mission  2  scores  from  VECTOR;  Final  mission  =  Change 
scores  for  Mission  1  and  Mission  2;  On  task  =  On  task  scale;  Off  task  =  Off  task  scale. 

N=  159. 
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Table  3.  Descriptive  Statistics  for  Each  Feedback  Condition 


Positive  Negative 


Measures 

M 

SD 

95%  Cl 

Skewness 

Kurtosis 

M 

SD 

95%  Cl 

Skewness 

Kurtosis 

Positive 

4.98 

1.01 

[4.75,5.21] 

-0.99 

0.56 

2.70 

1.36 

[2.40,  2.99] 

0.55 

-0.60 

Negative 

1.85 

0.93 

[1.64,2.07] 

1.47 

2.67 

4.15 

1.23 

[3.89,  4.42] 

-0.85 

0.50 

Frequent 

3.50 

1.53 

[3.14,3.85] 

0.01 

-1.21 

3.23 

1.31 

[2.95,3.51] 

0.04 

-0.94 

Infrequent 

3.32 

1.64 

[2.94,  3.70] 

0.19 

-1.25 

3.70 

1.54 

[3.36,4.03] 

-0.08 

-1.30 

Self-efficacy 

80.71 

11.07 

[78.14,  83.27] 

-1.01 

2.17 

80.50 

10.14 

[78.31,82.69] 

-0.43 

0.22 

FO 

5.01 

0.61 

[4.87,  5.15] 

-1.21 

2.20 

4.98 

0.53 

[4.86,  5.09] 

-0.60 

0.43 

Presence 

All 

1.38 

[3.95,4.59] 

-0.56 

0.00 

4.16 

1.25 

[3.89,  4.43] 

-0.51 

0.23 

Pretest 

26.62 

3.18 

[25.88,  27.36] 

-1.68 

4.81 

25.87 

3.71 

[25.07,  26.67] 

0.42 

3.42 

Posttest 

29.09 

1.81 

[28.67,  29.51] 

-1.50 

3.61 

28.67 

2.03 

[28.23,  29.10] 

-0.97 

1.09 

Final  test 

2.47 

3.07 

[1.76,3.18] 

1.68 

5.88 

2.80 

3.62 

[2.01,3.58] 

-0.87 

6.20 

Mission  1 

10.17 

1.26 

[9.88,  10.47] 

-0.09 

-0.97 

9.76 

1.10 

[9.53,  10.00] 

-0.12 

-0.16 

Mission  2 

10.22 

1.01 

[9.98,  10.45] 

-0.29 

-0.34 

10.10 

1.01 

[9.89,  10.32] 

-0.43 

0.26 

Final  mission 

0.04 

1.37 

[-0.28,  0.36] 

0.06 

-0.33 

0.34 

1.50 

[0.02,  0.66] 

-0.24 

0.52 

On  task 

5.93 

1.10 

[5.67,  6.18] 

-0.58 

-0.07 

5.74 

0.91 

[5.55,  5.94] 

-0.03 

-0.67 

Off  task 

2.72 

1.29 

[2.42,3.01] 

1.29 

3.12 

3.05 

1.24 

[2.79,  3.32] 

0.40 

-0.59 

Note.  Positive  =  Manipulation  check  scores  for  positive  feedback  condition;  Negative  =  Manipulation  check  scores  for  negative  feedback 
condition;  Frequent  =  Manipulation  check  scores  for  frequent  feedback  condition;  Infrequent  =  Manipulation  check  scores  for  infrequent 
feedback  condition;  Self-efficacy  =  Self-efficacy  scale;  FO  =  Feedback  orientation  scale;  Presence  =  Presence  scale;  Pretest  =  Pretest  scores  on 
VECTOR  questionnaire;  Posttest  =  Posttest  scores  on  VECTOR  questionnaire;  Final  test  =  Change  scores  for  pretest  and  posttest;  Mission  1  = 
Mission  1  scores  from  VECTOR;  Mission  2  =  Mission  2  scores  from  VECTOR;  Final  mission  =  Change  scores  for  Mission  1  and  Mission  2; 
On  task  =  On  task  scale;  Off  task  =  Off  task  scale. 

N=  159. 
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Table  3  (con’t).  Descriptive  Statistics  for  Each  Feedback  Condition 


Frequent  Infrequent 


Measures 

M 

SD 

95%  Cl 

Skewness 

Kurtosis 

M 

SD 

95%  Cl 

Skewness 

Kurtosis 

Positive 

3.67 

1.77 

[3.29,  4.06] 

-0.09 

-1.46 

3.85 

1.53 

[3.50,  4.21] 

-0.34 

-0.97 

Negative 

3.31 

1.70 

[2.94,  3.68] 

0.23 

-1.28 

2.83 

1.42 

[2.50,3.16] 

0.23 

-1.28 

Frequent 

4.26 

1.15 

[4.01,4.50] 

0.43 

-0.15 

2.35 

0.93 

[2.13,2.56] 

0.43 

-0.15 

Infrequent 

2.45 

1.12 

[2.21,2.70] 

0.71 

-0.19 

4.72 

1.13 

[4.46,4.98] 

-0.85 

0.14 

Self-efficacy 

79.92 

10.78 

[77.58,  82.26] 

-1.13 

2.20 

81.35 

10.3 

[78.98,  83.73] 

-0.22 

-0.22 

FO 

4.99 

0.53 

[4.88,5.11] 

-0.82 

0.57 

4.98 

0.61 

[4.85,5.12] 

-1.00 

1.99 

Presence 

4.36 

1.35 

[4.06,  4.65] 

-0.56 

-0.04 

4.05 

1.26 

[3.76,4.34] 

-0.56 

0.38 

Pretest 

25.48 

3.72 

[24.67,  26.28] 

-0.97 

0.89 

27.05 

3.02 

[26.36,  27.75] 

1.27 

7.26 

Posttest 

28.98 

1.80 

[28.59,  29.37] 

-1.16 

2.22 

28.73 

2.09 

[28.26,  29.21] 

-1.17 

1.57 

Final  test 

3.51 

3.45 

[2.76,  4.26] 

1.17 

1.90 

1.68 

3.03 

[0.98,2.38] 

-2.13 

2.22 

Mission  1 

9.93 

1.19 

[9.67,  10.19] 

0.01 

-0.59 

9.99 

1.20 

[9.71,  10.26] 

-0.07 

-0.53 

Mission  2 

10.08 

1.01 

[9.86,  10.30] 

-0.31 

0.42 

10.24 

1.01 

[10.00,  10.47] 

-0.42 

-0.44 

Final  mission 

0.15 

1.50 

[-1.17,0.48] 

0.06 

0.37 

0.25 

1.39 

[-0.07,  0.57] 

-0.28 

-0.61 

On  task 

5.81 

0.95 

[5.60,  6.02] 

-0.37 

-0.31 

5.38 

1.07 

[5.61,6.10] 

-0.28 

-0.39 

Off  task 

2.83 

1.20 

[2.57,  3.09] 

0.65 

-0.04 

2.97 

0.88 

[2.66,  3.28] 

0.88 

1.51 

Note.  Positive 

=  Manipulation  check  scores 

for  positive  feedback  condition;  Negative 

=  Manipulation  check  scores  for  negative  feedback 

condition;  Frequent  =  Manipulation  check  scores  for  frequent  feedback  condition;  Infrequent  =  Manipulation  check  scores  for  infrequent 
feedback  condition;  Self-efficacy  =  Self-efficacy  scale;  FO  =  Feedback  orientation  scale;  Presence  =  Presence  scale;  Pretest  =  Pretest  scores  on 
VECTOR  questionnaire;  Posttest  =  Posttest  scores  on  VECTOR  questionnaire;  Final  test  =  Change  scores  for  pretest  and  posttest;  Mission  1  = 
Mission  1  scores  from  VECTOR;  Mission  2  =  Mission  2  scores  from  VECTOR;  Final  mission  =  Change  scores  for  Mission  1  and  Mission  2; 
On  task  =  On  task  scale;  Off  task  =  Off  task  scale. 

N=  159. 
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Table  4.  Correlations  Among  Studied  Variables  -  Overall  Sample 


Measure 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 .  Positive 

2.  Negative 

-.810 

.... 

3.  Frequent 

.172* 

.040 

.... 

4.  Infrequent 

-.080 

-.026 

-.733** 

— 

5.  Self-efficacy 

.119 

-.074 

-.109 

.086 

— 

6.  FO 

.103 

-.115 

.021 

-.022 

.312** 

7.  Presence 

.124 

.027 

.113 

-.173* 

.145 

.308** 

.... 

8.  Pretest 

.089 

-.185* 

-.225** 

.103 

.172* 

.118 

.007 

— 

9.  Posttest 

.038 

-.044 

-.115 

-.013 

.028 

.091 

.166* 

.336** 

— 

10.  Final  test 

-.070 

.166* 

.166* 

-.114 

-.162* 

-.070 

.103 

_  841** 

727** 

.... 

11.  Mission  1 

.366** 

-.367** 

-.054 

.072 

.121 

2^7** 

.500 

.308** 

.199* 

_  204** 

12.  Mission  2 

.114 

. 242** 

-.088 

.079 

.030 

.183* 

.008 

.152 

.109 

-.095 

.148 

13.  Final 
mission 

_  772** 

.133 

-.017 

-.004 

-.079 

-.096 

-.035 

-.148 

-.088 

.103 

-.722** 

.577** 

14.  On  task 

.262** 

-.188* 

.008 

-.041 

219** 

42i** 

* 

m 

00 

-.020 

-.139 

-.060 

.075 

.102 

.009 

15.  Off  task 

-.204** 

.204** 

-.056 

.190* 

-.223** 

-.252** 

-.062 

-.159* 

-.172* 

.066 

-.297** 

.032 

.268** 

-.055 

.... 

Note.  For  all  scales,  higher  scores  are  indicative  of  more  extreme  responding  in  the  direction  of  the  construct  assessed.  Positive  =  Manipulation  check  scores 
for  positive  feedback  condition;  Negative  =  Manipulation  check  scores  for  negative  feedback  condition;  Frequent  =  Manipulation  check  scores  for  frequent 
feedback  condition;  Infrequent  =  Manipulation  check  scores  for  infrequent  feedback  condition;  Self-efficacy  =  Self-efficacy  scale;  FO  =  Feedback 
orientation  scale;  Presence  =  Presence  scale;  Pretest  =  Pretest  scores  on  VECTOR  questionnaire;  Posttest  =  Posttest  scores  on  VECTOR  questionnaire;  Final 
test  =  Change  scores  for  pretest  and  posttest;  Mission  1  =  Mission  1  scores  from  VECTOR;  Mission  2  =  Mission  2  scores  from  VECTOR;  Final  mission  = 
Change  scores  for  Mission  1  and  Mission  2;  On  task  =  On  task  scale;  Off  task  =  Off  task  scale. 

*p<  .05,  **<  .01. 
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Main  Results 


Hypotheses  1  through  4  were  examined  by  performing  a  2  x  2  mixed-design  multivariate 
analysis  of  variance  (MANOVA)  with  two  dependent  variables.  The  dependent  variables  (DV) 
were  the  change  scores  for  the  VECTOR  scenario  questionnaire  (pretest  and  posttest)  and  the 
change  score  for  Mission  1  and  Mission  2.  The  independent  variables  (IV)  were  between-subject 
variables  consisting  of  feedback  frequency  (frequent  and  infrequent)  and  feedback  sign  (positive 
and  negative). 

The  Wilks’  Lambda  omnibus  test  was  used  to  determine  whether  there  was  a  significant 
difference  in  at  least  one  of  the  dependent  variables.  The  results  showed  that  the  combined  DV’s 
were  significantly  affected  by  frequency  ( F  (2,  154)  =  6.28,  p  <  .00),  and  the  interaction  of 
frequency  and  sign  (. F  (2,  154)  =  3.20 ,p  =  .04),  but  not  by  sign  (F  (2,  154)  =  .79, p  =  .45). 
Feedback  frequency  explained  7.5%  of  the  variance  in  the  DV’s  and  the  interaction  explained 
4.0%  of  the  variance  in  the  DV’s.  The  association  was  less  substantial  for  the  nonsignificant 
main  effect  of  sign  explaining  1.0%  of  the  variance  in  the  DV’s. 

Hypothesis  1  stated  that  the  feedback  intervention  theory  would  be  supported  if  participants  in 
the  infrequent  feedback  group  showed  better  performance  than  trainees  in  the  frequent  feedback 
group.  Hypothesis  2  stated  that  the  ACT-R  theory  would  be  supported  if  participants  in  the 
frequent  feedback  group  had  better  performance  than  trainees  in  the  frequent  feedback  group. 

The  results  of  the  between-subject  effects  showed  that  there  was  a  significant  difference  between 
the  feedback  frequency  groups  for  the  pre  and  post-test  (F  (1,  155)  =  1 1.61  ,p<  .00,  rj  =  .07),  but 
not  the  mission  scores  (F  (1,  155)  =  .46,  p  =  .50,  tf  =  .003).  Participants  in  the  frequent  feedback 
condition  had  a  greater  improvement  in  post- test  scores  (M=  3.51)  than  those  in  the  infrequent 
feedback  condition  (M=  1.68),  supporting  Hypothesis  2  and  the  ACT-R  theory. 

Hypothesis  3  stated  that  the  social  cognitive  theory  would  be  supported  if  participants  in  the 
positive  feedback  group  performed  higher  than  participants  in  the  negative  feedback  group. 
Hypothesis  4  stated  that  control  theory  would  be  supported  if  individuals  in  the  negative 
feedback  group  performed  higher  than  individuals  in  the  positive  feedback  group.  The  Wilks’ 
lambda  criterion  indicated  there  was  no  significant  difference  between  the  feedback  sign  groups 
on  the  DVs  (F  (2,  154)  =  .79,  p  =  .45).  For  this  reason,  the  between-subject  effects  were  not 
further  evaluated.  Thus,  neither  Hypothesis  3  nor  Hypothesis  4  was  supported.  The  sign  of  the 
feedback  did  not  have  a  significant  impact  on  performance. 

Hypothesis  5  stated  that  feedback  orientation  would  moderate  the  relationship  between  feedback 
intervention  (feedback  sign  and  feedback  frequency)  and  performance  (pre  and  post-test  and 
mission  scores).  The  procedures  used  for  the  moderated  regression  analyses  were  those 
presented  in  Baron  and  Kenny  (1986).  The  following  hierarchical  regression  equations  were 
estimated:  (1)  regressing  the  dependent  variable  on  the  independent  variable,  (2)  regressing  the 
dependent  variable  on  the  moderator,  and  (3)  regressing  the  dependent  variable  on  the  product  of 
the  independent  variable  and  the  moderator. 

The  hypothesis  would  be  supported  if,  when  the  independent  variable  and  the  moderator  were 
controlled,  the  product  of  the  independent  variable  and  moderator  remains  significant.  The 
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results  of  the  moderated  regression  are  shown  in  Table  5  for  each  feedback  intervention  and 
dependent  variable.  Hypothesis  5  was  not  supported.  Feedback  orientation  was  not  found  to 
moderate  the  relationship  between  any  feedback  intervention  type  and  performance. 


Table  5.  Results  of  Hierarchical  Regression  Analyses  with  Feedback  Orientation  as  Moderator 


Predictor 

A/?2 

F 

P 

Feedback  Sign  (Positive/Negative) 

DV:  Final  Test 

Step  1:  Feedback  sign 

0.002 

0.363 

-0.048 

Step  2:  FO 

0.005 

0.551 

-0.069 

Step  3:  Feedback  sign*FO 

0.006 

0.662 

0.673 

DV:  Final  Mission 

Step  1:  Feedback  sign 

0.011 

1.722 

-0.104 

Step  2:  FO 

0.009 

1.558 

-0.093 

Step  3:Feedback  sign*FO 

0.000 

1.052 

0.175 

Feedback  Frequency  (Frequent/Infrequent) 

DV:  Final  Test 

Step  1:  Feedback  frequency 

0.074*** 

12.494*** 

0.272*** 

Step  2:  FO 

0.005 

6.690** 

-0.073 

Step  3:  Feedback  frequency  *  FO 

0.002 

4.545** 

-0.386 

DV:  Final  Mission 

Step  1:  Feedback  frequency 

0.001 

0.184 

-0.034 

Step  2:  FO 

0.009 

0.817 

-0.096 

Step  3:  Feedback  frequency  *  FO 

0.003 

0.679 

-0.640 

Note.  FO  =  Feedback  orientation;  Final  test  =  change  scores  for  pre  and  posttest; 
Final  mission  =  change  scores  for  Mission  1  and  Mission  2 
*p  <  0.05,  **p  <  0.01,  ***p  <  0.001 
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Supplementary  Results 


As  an  exploratory  approach,  the  interaction  between  feedback  frequency  and  feedback  sign  was 
examined.  The  Wilk’s  lambda  criterion  indicated  a  significant  interaction  between  feedback 
frequency  and  feedback  sign  (/?  =  .04,  if  =  .04).  Results  of  the  test  of  between-subject  effects 
indicated  there  was  a  significant  interaction  for  the  mission  scores  (F(  \ ,  155)  =  5.21,  p  =  .02,  if  = 
.03)  but  not  the  pretest  and  posttest  scores  (F(l,155)  =  1.77,/?  =  .19).  The  plot  of  the  interaction 
for  the  mission  scores  is  presented  in  Figure  2.  Post-hoc  independent  sample  t-tests  were 
conducted  to  determine  which  groups  were  significantly  different  within  the  mission  scores.  The 
results  indicate  no  significant  differences  between  infrequent  positive  feedback  (M=  .37)  and 
infrequent  negative  feedback  (M  =-.13,/=  .73,  p  =  .47).  However,  a  significant  difference  was 
found  between  frequent  positive  feedback  (M=  -.30)  and  frequent  negative  feedback  (M=  .50,  t 
=  -2.51,/?  =  .01).  Individuals  receiving  frequent  negative  feedback  had  better  performance  than 
those  with  infrequent  positive  feedback.  Results  from  the  t-test  also  showed  a  significant 
difference  between  frequent  positive  feedback  (M=  -.30)  and  infrequent  positive  feedback  (M  = 
.37,  t  =  -2.17,/?  =  .03)  with  infrequent  positive  feedback  leading  to  better  performance.  No 
differences  were  found  between  frequent  negative  feedback  (M=  .50)  and  infrequent  negative 
feedback  (M=  .13,  t  =  1.11,/?=  .27). 

A  moderated  regression  analysis  was  also  conducted  to  further  explore  the  relationship  between 
feedback  intervention  and  performance.  Moderated  regression  analyses  were  conducted  to 
determine  if  self-efficacy,  on-task  attention,  off-task  attention  or  presence  moderated  the 
relationship  between  feedback  intervention  and  perfonnance.  The  results  of  the  moderated 
regression  analyses  are  found  in  Table  A6.  None  of  the  variables  tested  were  moderators. 
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Figure  2.  Effects  of  feedback  frequency  and  sign  on  mission  difference  scores. 


Table  6.  Results  of  Exploratory  Hierarchical  Regression  Analyses  with  Self-Efficacy,  On-Task,  Off-Task 


and  Presence  as  Moderator 


Moderators 

AR2 

F 

P 

Self-Efficacy 

Feedback  Sign  (Positive/Negative) 

DV:  Final  Test 

Step  1:  Feedback  sign 

0.002 

0.363 

-0.048 

Step  2:  Self-efficacy 

0.026* 

2.279 

-0.161 

Step  3:  Feedback  sign*Self-efficacy 

DV:  Final  Mission 

0.004 

1.715 

-0.476 

Step  1:  Feedback  sign 

0.011 

1.722 

-0.104 

Step  2:  Self-efficacy 

0.006 

1.342 

-0.078 

Step  3:Feedback  sign*Self-efficacy 

0.011 

1.459 

0.800 

Feedback  Frequency  (Frequent/Infrequent) 

DV:  Final  Test 

Step  1:  Feedback  frequency 

0.074*** 

12.494*** 

0.272*** 

Step  2:  Self-efficacy 

0.021 

8.130*** 

-0.144 

Step  3:  Feedback  frequency  *  Self-efficacy 

DV:  Final  Mission 

0.003 

5.592*** 

-0.445 

Step  1:  Feedback  frequency 

0.001 

0.184 

-0.034 

Step  2:  Self-efficacy 

0.007 

0.613 

-0.082 

Step  3:  Feedback  frequency  *  Self-efficacy 

0.001 

0.472 

0.276 

On-task 

Feedback  Sign  (Positive/Negative) 

DV:  Final  Test 

Step  1:  Feedback  sign 

0.002 

0.363 

-0.048 

Step  2:  On-task 

0.003 

0.422 

-0.694 

Step  3:  Feedback  sign*On-task 

DV:  Final  Mission 

0.002 

0.373 

0.261 

Step  1:  Feedback  sign 

0.011 

1.722 

-0.104 

Step  2:  On-task 

0.000 

0.883 

0.018 

Step  3:Feedback  sign*On-task 

0.002 

0.680 

0.261 

Feedback  Frequency  (Frequent/Infrequent) 

DV:  Final  Test 

Step  1:  Feedback  frequency 

0.074*** 

12.494*** 

0.272*** 

Step  2:  On-task 

0.003 

6.472** 

-0.054 

Step  3:  Feedback  frequency  *  On-task 

DV:  Final  Mission 

0.004 

4.513** 

0.363 

Step  1:  Feedback  frequency 

0.001 

0.184 

-0.034 

Step  2:  On-task 

0.000 

0.096 

0.008 

Step  3:  Feedback  frequency  *  On-task 

0.002 

0.157 

0.252 

*p  <  0.05,  **p  <  0.01,  ***p  <  0.001 
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Table  6  (  Cont'd).  Results  of  Exploratory  Hierarchical  Regression  Analyses  with  Self-Efficacy,  On- 
Task,  Off-Task  and  Presence  as  Moderator 


Moderators 

A/?2 

F 

P 

Off-Task 

Feedback  Sign  (Positive/Negative) 

DV:  Final  Test 

Step  1:  Feedback  sign 

0.002 

0.363 

-0.048 

Step  2:  Off-task 

0.004 

0.461 

0.060 

Step  3:  Feedback  sign*Off-task 

DV:  Final  Mission 

0.000 

0.311 

-0.025 

Step  1:  Feedback  sign 

0.011 

1.722 

-0.104 

Step  2:  Off-task 

0.066*** 

6.479** 

0.259*** 

Step  3:Feedback  sign*Off-task 

0.004 

4.554** 

-0.169 

Feedback  Frequency  (Frequent/Infrequent) 

DV:  Final  Test 

Step  1:  Feedback  frequency 

0.074*** 

12.494*** 

0.272*** 

Step  2:  Off-task 

0.007 

6.804*** 

0.081 

Step  3:  Feedback  frequency  *  Off-task 

DV:  Final  Mission 

0.008 

4.997** 

0.235 

Step  1:  Feedback  frequency 

0.001 

0.184 

-0.034 

Step  2:  Off-task 

0.071*** 

6.080** 

0.267*** 

Step  3:  Feedback  frequency  *  Off-task 

0.000 

4.043** 

0.043 

Presence 

Feedback  Sign  (Positive/Negative) 

DV:  Final  Test 

Step  1:  Feedback  sign 

0.002 

0.363 

-0.048 

Step  2:  Presence 

0.011 

1.047 

0.105 

Step  3:  Feedback  sign*Presence 

DV:  Final  Mission 

0.004 

0.900 

-0.222 

Step  1:  Feedback  sign 

0.011 

1.722 

-0.104 

Step  2:  Presence 

0.001 

0.933 

-0.031 

Step  3:Feedback  sign*Presence 

0.001 

0.684 

-0.126 

Feedback  Frequency  (Frequent/Infrequent) 

DV:  Final  Test 

Step  1:  Feedback  frequency 

0.074*** 

12.494*** 

0.272*** 

Step  2:  Presence 

0.005 

6.672** 

0.072 

Step  3:  Feedback  frequency  *  Presence 

DV:  Final  Mission 

0.001 

4.465** 

-0.100 

Step  1:  Feedback  frequency 

0.001 

0.184 

-0.034 

Step  2:  Presence 

0.001 

0.168 

-0.032 

Step  3:  Feedback  frequency  *  Presence 

0.000 

0.122 

-0.053 

*p  <  0.05,  **p  <  0.01,  ***p  <  0.001 
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Discussion 


The  role  of  feedback  in  videogame-based  adaptive  training  is  not  well  understood.  The 
theoretical  accounts  of  the  impact  of  feedback  in  training  or  learning  a  new  skill  are 
contradictory  and  therefore  are  not  frequently  used  to  inform  the  design  of  training  programs. 

The  purpose  of  this  research  was  to  test  four  competing  feedback  theories  to  determine  which 
type  of  feedback  would  lead  to  better  perfonnance  in  a  videogame-based  adaptive  training 
environment.  It  is  hoped  that  ultimately  the  results  of  this  research  can  be  used  by  videogame 
simulation  designers  to  design  training  and  include  appropriate  performance  feedback. 

The  first  set  of  hypotheses  tested  competing  feedback  frequency  theories.  According  to  the 
feedback  intervention  theory,  feedback  should  be  infrequent  during  a  complex  task  because 
feedback  may  overload  cognitive  capacity  at  a  time  when  much  of  the  learner’s  attention  is 
needed  (Kluger  &  DeNisi,  1996).  Contrary  to  this  perspective,  the  ACT-R  model  suggests  that 
frequent  feedback  is  essential  for  acquiring  any  skill  because  infrequent  feedback  slows  down 
the  process  of  compiling  correct  infonnation.  The  results  of  the  main  effect  for  frequency 
reinforce  Anderson’s  ACT-R  model.  That  is,  in  videogame-based  training  scenarios,  frequent 
feedback  leads  to  higher  post  test  scores  than  infrequent  feedback.  It  appears  that  infrequent 
feedback  did  not  provide  enough  cues  for  the  participant  to  detect  and  reject  erroneous 
hypotheses  and  this  in  turn  caused  the  participant  to  compile  incorrect  information  which 
decreased  perfonnance  (Anderson,  1983,  1996,  2000).  Complex  tasks,  such  as  the  scenarios 
presented  in  this  research,  may  include  rapid  changes  occurring  between  learning  episodes, 
which  offers  several  opportunities  for  unmonitored  mistakes.  Thus,  a  long  pause  between 
feedback  interventions  may  lead  to  learning  the  task  incorrectly  and  later  applying  this  incorrect 
learning  structure  to  future  problems. 

Although  no  support  was  found  for  the  feedback  intervention  theory  this  may  be  due  to  the  lack 
of  focus  in  the  theory  on  the  individual’s  learning  stage.  The  current  investigation  examined  a 
novel,  complex  task,  which  may  reflect  the  beginning  stages  of  learning  because  the  participants 
did  not  have  previous  experience  learning  complex  interpersonal  skills  in  an  adaptive 
videogame -based  environment.  It  is  possible  that  while  frequent  feedback  is  useful  during  the 
initial  stages  of  learning  (declarative  and  procedural)  this  feedback  can  become  detrimental  as 
the  individual  becomes  more  proficient  in  the  task.  Future  research  should  examine  the 
possibility  of  this  relationship. 

Competing  feedback  sign  theories  were  also  examined.  Self-efficacy  theory  suggests  that  people 
avoid  activities  that  they  believe  are  beyond  their  coping  capabilities,  but  they  undertake  and 
perform  activities  they  judge  themselves  capable  of  managing.  Therefore,  receiving  positive 
feedback  is  more  likely  to  raise  self-efficacy  and  in  turn  increase  perfonnance  as  opposed  to 
negative  feedback.  On  the  other  hand,  control  theory  recommends  the  use  of  negative  feedback 
because  it  is  more  likely  to  interrupt  processing,  which  will  shift  attention  and  improve 
performance.  The  results  of  this  research  show  no  support  for  the  main  effect  of  feedback  sign 
on  perfonnance,  and  thus  neither  theory  regarding  the  role  of  feedback  sign  on  training 
performance  was  supported.  This  is  in  line  with  previous  studies  that  have  found  it  difficult  to 
predict  the  effects  of  feedback  sign  on  performance  (Kluger  &  DeNisi,  1996).  In  their  meta- 
analytic  study  Kluger  and  DeNisi  (1996)  found  that  feedback  sign  did  not  impact  the  relationship 
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between  feedback  and  performance  on  a  task.  The  authors  suggest  these  results  are  due  to  the 
difficulty  in  predicting  a  person’s  response  to  feedback,  particularly  negative  feedback,  without 
knowing  other  factors  about  the  person.  It  is  likely  that  the  outcome  of  feedback  sign  is  more 
appropriately  thought  of  as  a  complex  interaction  among  feedback  sign,  situational  factors  and 
individual  dispositions. 

Another  possible  reason  for  the  lack  of  effect  of  feedback  sign  on  performance  may  be  the  type 
of  task.  Previous  studies  have  examined  feedback  sign  in  a  face-to-face  environment  such  as 
receiving  positive  or  negative  feedback  from  a  supervisor  (Becker  &  Kilmoski,  1989; 
Chakrabarty  et  ah,  2008;  Jaworski  &  Kohli,  1991).  However,  feedback  sign  in  videogame-based 
adaptive  training  may  not  have  the  same  influence  as  other  tasks.  In  fact,  previous  research  has 
found  an  interactive  effect  of  feedback  sign  and  task  type  on  motivation  and  performance 
(Kluger  &  DeNisi,  1996,  2005;  Van-Dijk  &  Kluger,  2004).  An  individual  completing  a 
computer-based  task  may  find  negative  feedback  more  readily  acceptable  when  it  is  received 
from  the  computer  rather  than  face-to-face  from  a  supervisor.  Earley  (1988)  found  the  source  of 
the  feedback,  person  versus  a  computer,  was  directly  related  to  performance.  Feedback  had  a 
greater  impact  on  an  individual’s  perfonnance  if  the  feedback  was  provided  directly  from  the 
computer  system  than  if  provided  by  a  supervisor.  Similarly,  Kluger  and  Adler  (1993)  found 
participants  were  more  likely  to  seek  feedback  from  a  computer  rather  than  a  person.  Therefore, 
task  type  and  source  may  play  a  larger  role  in  predicting  perfonnance  than  the  sign  of  the 
feedback.  Research  should  continue  to  examine  the  role  of  computer-mediated  feedback  in 
learning  and  development. 

Further,  in  the  videogame-based  adaptive  training  environment  participants  may  have  received 
all  the  positive  and  negative  feedback  they  needed  through  the  reactions  of  the  characters, 
making  the  mission  feedback  irrelevant.  Kluger  and  DeNisi  (19996)  advise  that  even  if  the 
feedback  is  directed  towards  the  task,  if  it  is  redundant  with  the  preexisting  knowledge  it  will 
have  no  effect  on  learning. 

The  interaction  between  feedback  frequency  and  feedback  sign  was  also  examined.  The 
interaction  was  significant  and  further  post-hoc  analyses  indicated  that  frequent  feedback  was 
most  beneficial  when  the  feedback  was  negative  rather  than  positive.  Also,  positive  feedback 
leads  to  better  performance  when  it  was  infrequent  rather  than  frequent.  Consistent  with  these 
results,  previous  studies  have  shown  support  for  the  use  of  infrequent  positive  feedback  instead 
of  frequent  positive  feedback  (Earley,  Connolly  &  Ekegren,  1989;  Baumeister,  Hutton  &  Caims, 
1990;  Kluger  &  DeNisi,  1996).  According  to  the  feedback  intervention  theory,  frequent 
feedback  that  does  not  give  details  on  improvement  (e.g.  praise)  can  distract  the  individual  from 
the  task  and  bring  attention  to  the  self.  Attention  to  the  self  results  in  superior  performance  only 
if  the  task  is  very  simple  (Kluger  &  DeNisi,  1996).  Similarly,  it  has  been  suggested  that  being 
overly  confident  about  the  task,  which  can  be  induced  through  frequent  positive  feedback,  may 
lead  to  experimenting  with  task  strategies  and  in  turn  lead  to  poor  perfonnance  (Earley,  Connolly 
&  Ekegren,  1989).  Indeed,  results  of  a  study  by  Earley,  Connolly,  and  Ekegren  (1989)  showed 
that  an  increase  in  motivation  leads  to  an  increase  in  dysfunctional  strategy  search.  Additionally, 
Baumeister,  Hutton  and  Cairns  (1990)  found  that  praise  (a  type  of  positive  feedback 
intervention)  impaired  the  perfonnance  of  a  cognitively  demanding  task  but  improved  the 
performance  of  a  simple  task.  Stone  (1994)  also  found  that  high  self-efficacy,  which  is  induced 
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through  frequent  nonnative  feedback,  led  to  overconfidence  in  one’s  ability.  As  a  result, 
individuals  with  high  self-efficacy  contributed  fewer  resources  toward  the  task.  Overall, 
previous  research  suggests  that  frequent  positive  feedback  can  be  detrimental  to  performance 
during  a  complex  task.  In  line  with  previous  research,  the  results  of  this  investigation  show  that 
frequent  positive  feedback  leads  to  poor  performance  while  infrequent  positive  feedback  leads  to 
better  perfonnance.  Given  these  findings,  it  appears  that  infrequent  positive  feedback  serves  as  a 
motivator  but  because  it  is  infrequent  it  will  not  distract  the  individual  by  bringing  attention  to 
the  self  instead  of  the  task. 

This  investigation  also  suggests  that  frequent  negative  feedback  leads  to  the  best  performance  in 
mission  scores.  This  is  in  line  with  the  error  management  training  literature  (Aguinis  &  Kraiger, 
2009;  Keith  &  Frese,  2005).  Error  management  training  encourages  trainees  to  make  errors  and 
engage  in  reflection  to  understand  why  the  errors  occurred.  Additionally,  error  management 
training  encourages  trainees  to  leam  how  to  develop  new  strategies  to  avoid  repeating  the  same 
errors  in  the  future.  Error  management  training  is  often  duplicated  by  giving  participants 
frequent  negative  feedback  (Aguinis  &  Kraiger,  2009).  A  meta-analysis  conducted  by  Keith  and 
Frese  (2005)  reported  that  error  management  training  lead  to  better  performance  in  comparison 
to  proceduralized  error-avoidant  training  and  exploratory  training  without  error  encouragement. 
Thus,  frequent  negative  feedback  may  lead  to  a  deeper  understanding  of  the  task  as  suggested  by 
the  error  management  literature.  Additionally,  the  improvement  in  performance  for  participants 
in  the  frequent  negative  feedback  condition  suggests  support  for  control  theory.  Negative 
feedback  results  in  movement  toward  the  standard  and  the  more  instances  of  the  negative 
feedback,  the  more  chances  for  that  movement.  Thus,  the  control  theory  suggests  that  frequent 
negative  feedback  leads  to  perfonnance  improvement.  Due  to  the  exploratory  nature  of  this 
analysis,  future  research  should  further  examine  the  possible  relationship  between  feedback 
frequency  and  feedback  sign. 

In  addition  to  feedback  frequency  and  feedback  sign,  it  was  predicted  that  feedback  orientation 
would  moderate  the  relationship  between  feedback  intervention  and  performance  such  that  those 
with  high  feedback  orientation  would  perform  better  with  frequent  feedback  and  those  with  low 
feedback  orientation  would  perform  better  with  infrequent  feedback.  Furthermore,  individuals 
with  low  feedback  orientation  would  perfonn  better  with  positive  feedback  while  those  with  high 
feedback  orientation  would  perform  the  same  regardless  of  feedback  sign.  This  prediction  was 
not  supported.  In  a  training  environment,  feedback  orientation  may  be  less  influential  because 
trainees  are  expecting  to  receive  feedback  on  their  performance.  On  the  other  hand,  the 
influence  of  feedback  orientation  may  be  different  if  examined  in  a  situation  such  as  daily 
performance  on  the  job.  In  this  situation  individuals  may  not  be  expecting  feedback  from  their 
supervisor  or  co-workers  and  thus  how  they  interpret  and  use  the  feedback  may  have  a  larger 
influence.  This  is  the  first  effort  to  look  at  feedback  orientation  within  a  training  context; 
previous  studies  focusing  on  feedback  orientation  have  utilized  survey  designs  (Linderbaum  & 
Levy,  2007).  While  feedback  orientation  did  not  moderate  as  predicted,  it  did  have  a  significant 
positive  correlation  with  self-efficacy  for  learning  interpersonal  skills.  This  builds  on 
Linderbaum  and  Levy  (2007)  who  found  general  self-efficacy  to  be  positively  correlated  to 
feedback  orientation. 
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Exploratory  analyses  were  conducted  around  the  other  individual  difference  variables  (self- 
efficacy,  presence  and  attention  to  the  task)  to  examine  whether  they  interacted  with  feedback 
intervention  to  impact  performance.  These  variables  were  not  supported  as  moderators.  These 
results  may  be  due  to  the  short  time-frame  of  the  interpersonal  skills  training  and  the  research 
design  (between-subjects  design).  It  is  possible  that  during  training  with  longer  duration, 
individual  differences  may  have  a  stronger  impact  and  play  a  significant  role  between  feedback 
intervention  and  performance.  In  fact,  there  has  been  a  debate  regarding  the  use  of  between- 
subject  design  and  within-subject  design  to  examine  the  influence  of  self-efficacy  on 
performance.  Vancouver,  Thompson  and  Williams  (2001)  examined  the  influence  of  self- 
efficacy  on  performance  as  a  between-person  and  within-person  approach.  Participants  in  the 
study  were  asked  to  take  part  in  an  analytical  game  called  Mastennind.  Results  indicate  that 
high  self-efficacy  lead  to  overconfidence  and  decreased  performance  in  a  within-person  design. 
Also,  the  study  found  significant  positive  between-person  correlation  between  self-efficacy  and 
performance.  Vancouver,  Thompson,  Tischner  &  Putka,  2002;  Vancouver  and  Kendall  (2006) 
replicated  the  previous  study  using  the  Mastermind  task  and  found  similar  results.  At  the  within- 
person  level,  the  manipulation  increased  self-efficacy  but  decreased  subsequent  performance.  At 
the  between-person  level,  self-efficacy  had  no  overall  relation  to  perfonnance.  Richard, 
Dieffendorf  and  Martin  (2006)  replicated  these  studies  using  different  tasks,  exam  performance 
in  a  classroom  context  and  a  computerized  learning  task  in  a  lab  setting.  The  researchers  found 
the  same  relationships  between  self-efficacy  and  performance.  Overall,  the  findings  of  these 
studies  suggest  that  a  longitudinal,  within-person  design  is  necessary  to  properly  assess  the 
direction  of  causality  for  self-efficacy  and  performance.  This  can  help  explain  why  the  present 
research  did  not  find  significant  correlations  between  self-efficacy,  or  any  of  the  individual 
difference  variables,  and  performance. 

There  were,  however,  interesting  significant  correlations  among  the  individual  difference 
variables  that  should  be  noted.  Pre-test  scores  and  self-efficacy  were  positively  correlated.  This 
provides  some  support  for  the  usefulness  of  the  pre-test  in  that  those  who  reported  higher 
interpersonal  skills  self  confidence  performed  better  on  a  test  of  interpersonal  skills.  Also,  self- 
efficacy  was  positively  correlated  with  reporting  of  being  on-task  (i.e.  paying  attention  to  the 
task)  during  the  training.  On  the  other  hand,  self-efficacy  was  negatively  correlated  to  off-task 
attention.  These  results  suggest  that  individuals  with  higher  self-efficacy  were  probably  more 
interested  and  motivated  by  the  task,  which  may  have  lead  them  to  pay  more  attention.  The 
lower  an  individual’s  self-efficacy  the  more  they  reported  being  off-task  (i.e.  not  paying  attention 
to  the  task)  during  the  training.  Being  off-task  was  also  negatively  correlated  with  pre  and  post¬ 
test  scores,  which  is  expected  because  attention  to  task  usually  relates  to  successful  task 
performance.  Lastly,  presence  and  post-test  scores  were  positively  correlated.  Individuals  who 
felt  immersed  in  the  game  performed  better  on  the  post-test. 

Finally,  it  should  be  noted  that  there  were  different  patterns  in  results  for  the  pretest/posttest  and 
Mission l/Mission2  scores.  There  was  a  significant  main  effect  for  the  test  scores;  however  there 
was  a  significant  interaction  for  the  mission  scores.  It  was  first  theorized  that  the  two  measures 
both  assessed  the  domains  of  interpersonal  skills  but  using  different  methods.  However,  the 
different  patterns  may  be  better  explained  using  Kraiger,  Ford,  and  Salas  (1993)  classification 
scheme  for  learning  outcomes.  In  their  model,  learning  may  be  evaluated  as  three  forms  of 
outcomes.  One  outcome  is  cognitive,  which  includes  verbal  knowledge,  knowledge 
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organization,  and  cognitive  strategies.  Another  outcome  is  skill-based,  which  includes 
compilation  of  information  and  automaticity  of  information.  The  third  outcome  is  affective 
involving  attitudinal  and  motivational  learning.  In  line  with  this  classification  scheme,  the 
current  research  may  have  been  tapping  into  knowledge  organization  and  cognitive  strategies 
(i.e.,  a  type  of  cognitive  outcome)  through  the  pretest  and  posttest,  which  is  paper-based  with 
simplified  scenarios.  On  the  other  hand,  the  mission  scores  may  be  evaluating  proceduralization 
(i.e.,  a  skill-based  outcome)  because  they  are  actually  learning  how  to  perform  the  task  in 
realistic  scenarios  while  receiving  feedback.  The  differences  between  these  measures  of 
performance  should  be  further  examined  in  future  research. 

Implications  and  Future  Research 

Ultimately,  the  aim  of  this  effort  was  to  begin  bridging  the  gap  in  the  feedback  and  training 
literature  to  help  researchers  revise  current  theoretical  frameworks  and  assist  practitioners  in 
developing  more  effective  training  environments.  The  results  reinforce  the  need  to  develop 
theories  that  take  into  consideration  both  feedback  frequency  and  feedback  sign  when  explaining 
learning  and  performance.  Researchers  in  the  industrial  organizational  psychology  field  should 
borrow  from  educational  psychology  and  consider  how  the  current  theories  would  apply  to 
different  stages  of  learning.  The  influence  of  feedback  type  may  vary  depending  on  the  learning 
stage  of  the  individual.  This  possibility  requires  further  investigation. 

Future  research  should  also  replicate  the  current  investigation  using  a  different  type  of  complex 
task  such  as  problem  solving  or  using  a  different  videogame-based  adaptive  training. 

Replicating  this  investigation  can  help  in  extending  the  generalizability  of  the  results  and  the 
theoretical  framework.  Additionally,  researchers  should  examine  and  test  the  interaction 
between  feedback  frequency  and  feedback  sign  in  more  detail  because  this  is  a  poorly  defined 
relationship. 

In  addition  to  the  theoretical  contributions  of  this  effort,  there  are  also  implications  for 
practitioners  working  with  videogame-based  adaptive  training.  Instructional  designers  can  use 
this  information  to  detennine  what  form  of  feedback  is  most  beneficial  to  training  a  complex 
task.  Most  guidelines  for  training  are  based  on  commonly  held  beliefs  instead  of  research.  In  the 
Training  Multimedia  Courseware  Development  Guide  provided  by  TRADOC  (“TRADOC 
Pamphlet  350-7-2”,  2003)  the  guidelines  suggest  that  only  positive  feedback  be  provided 
because  it  will  build  confidence  while  negative  feedback  will  discourage  the  student  and  should 
not  be  used.  It  also  suggests  that  positive  feedback  enhances  learning  and  builds  the  learner’s 
self-esteem  and  provides  motivation  to  leam.  However,  this  was  not  supported  in  the  current 
research.  Instead,  instructional  designers  need  to  keep  in  mind  that  during  a  complex  task  the 
trainee  requires  frequent  feedback  about  their  performance.  Preferably,  the  feedback  provided 
will  be  constructive  rather  than  just  positive  praise  and  adapt  to  the  trainee’s  learning  and 
performance.  The  use  of  technology  such  as  computer-based  training,  videogames  and  web- 
enabled  training  is  still  an  emerging  field  where  there  is  a  lack  of  theoretical  background  to  help 
develop  training.  This  effort  takes  a  step  toward  resolving  this  problem  to  allow  practitioners  a 
framework  upon  which  to  base  their  training  design. 
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Limitations 


There  are  certain  limitations  that  should  be  considered  when  interpreting  the  results.  First,  like 
many  studies,  the  sample  used  was  a  convenience  sample  from  two  universities  with  ages 
ranging  from  18  to  31.  This  is  not  a  representative  sample  of  the  population  and  thus  raises  the 
question  of  generalizability. 

Another  limitation  is  the  use  of  difference  scores  as  dependent  variables.  It  has  been  suggested 
that  there  are  methodological  flaws  that  need  to  be  considered  when  using  difference  scores. 
Some  of  the  problems  involve  the  validity  of  difference  scores,  such  as  ambiguous  interpretation 
and  spurious  correlations  (Edwards,  1994).  However,  it  has  been  suggested  that  the  use  of 
difference  scores  is  less  problematic  when  used  as  a  dependent  variables  instead  of  a  within- 
subjects  independent  variables  with  two  levels  (Edwards,  1994,  2001). 

An  additional  limitation  is  that  the  self-report  measures  used  were  developed  for  this  research 
and  had  not  been  previously  validated.  This  is  also  the  case  for  one  of  the  perfonnance 
measures,  the  pretest  and  posttest.  Although  this  is  a  limitation,  care  was  taken  to  develop  the 
scales  with  SME  input  and  appropriate  scale  development  guidelines  such  as  the  procedures 
outlined  by  Bandura  (2006). 

Also,  there  is  a  threat  to  internal  validity  that  should  be  considered  when  interpreting  the  results. 
More  specifically,  testing  is  an  issue  in  the  current  research  in  that  repeatedly  measuring  the 
participants  on  the  pretest  and  posttest  may  lead  to  bias.  This  is  especially  the  case  in  this 
research  because  the  time  lapse  between  the  pretest  and  the  posttest  was  usually  around  2  hours, 
which  may  not  be  a  significant  amount  of  time  between  the  two  measurements  to  prevent  a 
testing  threat.  Future  research  should  extend  the  current  work  using  a  larger  time  lapse  between 
the  measurements  to  detennine  if  this  is  a  major  threat. 

There  is  also  a  limitation  due  to  the  videogame.  During  the  game,  when  participants  interacted 
with  a  town  resident,  the  videogame  did  not  lock  the  view  on  the  character’s  face.  Depending  on 
where  the  avatar  was  placed,  some  participants  were  able  to  see  the  facial  reactions  of  the 
characters  but  other  participants  only  heard  the  individuals.  It  is  not  clear  whether  this  difference 
influenced  how  the  trainees  interpreted  the  feedback  or  how  this  may  have  influenced  the 
interactions  with  the  town  residents. 

Furthermore,  VECTOR  was  originally  developed  for  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  and  because  of  this  it  contains  military  content.  Trainees  are 
identified  as  Soldiers;  they  are  given  two  missions  in  a  Middle  Eastern-like  setting  and  they  have 
to  report  to  another  Soldier.  To  make  it  possible  for  university  students  to  play  the  game  a  lot  of 
the  military  content  was  removed.  Participants  did  not  require  previous  knowledge  of  military 
tactics  and  procedures  to  play  the  game.  It  is  not  known  if  a  Soldier  would  receive  different 
scores  on  this  game,  although  this  is  not  expected. 

Another  issue  with  using  VECTOR  is  the  difficulty  in  determining  whether  the  results  are  due  to 
the  specific  type  of  game  or  if  there  would  be  similar  results  using  other  computer-based 
training.  In  other  words,  would  these  results  transfer  to  another  form  of  computer-based 
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training?  Some  gaming  research  has  found  that  previous  experience  with  general  videogames 
(regardless  of  the  type  of  videogame  previously  played)  was  related  to  future  performance  in 
videogame -based  environments  (Gagnon,  1985;  Young,  Broach,  &  Farmer,  1997).  However, 
specific  prior  videogame  experience  that  share  similar  characteristics  with  the  training 
environment  can  provide  incremental  validity  over  general  experience  in  predicting  learning 
outcomes  (Orvis,  2005).  Therefore,  these  studies  suggest  that  the  learning  that  occurs  during  a 
specific  type  of  videogame-based  training  can  transfer  to  other  forms  of  videogame-based 
training  to  influence  performance.  Additionally,  research  has  shown  that  skills  learned  in  game- 
based  training  environments  transfer  to  real-life  situations  (Gopher,  Weil,  &  Baraket,  1994; 
Knerr,  Simutis,  &  Johnson,  1979).  Thus,  there  is  some  support  that  the  results  of  the  current 
research  would  be  similar  in  other  forms  of  videogame-based  training  and  real-life  situations. 

Conclusion 

Videogame-based  adaptive  training  has  provided  flexibility  and  adaptability  for  training  in  cost- 
effective  ways.  The  military  is  quickly  implementing  this  training  technique  without  a 
theoretical  background  that  explains  the  proper  design  or  methods  necessary  to  improve 
performance.  Clearly,  further  work  is  required  to  build  a  theoretical  framework  that  can  help 
explain  what  type  of  feedback  is  most  beneficial  to  leam  a  complex  task  in  a  videogame-based 
adaptive  environment.  However,  this  effort  serves  as  a  stepping  stone  towards  this  goal  by 
testing  four  competing  feedback  theories.  This  was  done  by  examining  the  role  of  feedback 
frequency  and  feedback  sign  on  performance.  Additionally,  feedback  orientation  was  examined 
to  determine  whether  it  moderates  the  relationship  between  feedback  intervention  and 
performance.  Although  not  all  predictions  were  supported,  frequent  feedback,  frequent  negative 
feedback  and  infrequent  positive  feedback  were  found  to  be  beneficial  to  trainees  while  learning 
a  complex  task  in  a  videogame-based  adaptive  training.  These  findings  serve  as  a  framework  for 
practitioners  in  determining  the  necessary  type  of  feedback  needed  when  designing  training  in 
the  emerging  field  of  videogame-based  adaptive  training.  Future  research  should  further 
examine  the  interaction  between  feedback  frequency  and  feedback  sign  and  integrate  those 
results  into  the  current  theoretical  perspectives.  In  the  end,  the  goal  of  this  effort  was  to  provide 
a  deeper  understanding  of  feedback’s  role  in  a  videogame -based  adaptive  training  environment. 
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Appendix 
Survey  Measures 

VECTOR  Scenario  Questionnaire  (Pre-Post  Test) 

You  will  be  presented  with  several  short  scenarios,  followed  by  several  possible  actions.  For 
each  scenario,  choose  the  action  you  feel  is  the  most  appropriate.  Circle  the  letter  to  indicate 
your  choice.  For  the  following  scenarios  you  are  an  Anny  Soldier  stationed  overseas. 


1 .  You  were  tasked  with  building  a  water  treatment  facility  in  a  small  town  and  maintaining  the 
peace.  In  talking  with  the  local  cleric  who  basically  runs  the  town,  he  has  agreed  to  let  you 
build  the  facility,  provided  some  of  his  men  are  paid  to  help  build  it.  You  are  not  sure  his 
men  have  any  experience  in  construction,  and  you  are  already  working  on  a  tight  budget. 
Choose  the  action  you  would  take. 


a. 

Replace  your  contractors  with  the  local  workers  to  please  the 
local  cleric. 

b. 

Wait  until  the  cleric  changes  his  mind  and  no  longer  wants  you 
to  hire  his  men.  Then  build  the  facility. 

c. 

Explain  you  cannot  afford  to  pay  his  men  and  build  the  facility 
using  your  contractors. 

d. 

Allow  a  few  men  to  work  but  request  they  bring  a  few  tools  to 
help  off-set  the  cost. 

2.  The  town  in  Lieutenant  Colonel  Bateman’s  area  of  operations  has  been  relatively  calm  after 
several  arguments  have  occurred  between  the  townspeople  and  the  government  concerning 
the  operation  of  the  new  hospital.  Lieutenant  Colonel  Bateman  has  just  arrived  for  a  meeting 
with  Hamad  to  discuss  the  current  state  of  the  hospital.  They  exchange  greetings  at  the  door 
and  walk  towards  a  table.  Before  the  discussion  begins,  what  should  Lieutenant  Bateman 
make  sure  he  does? 


a. 

Remove  body  armor  and  helmet. 

b. 

Get  right  down  to  business  and  question  him  over  what  he  knows 
about  the  hospital;  avoid  boring  Hamad  with  small  talk. 

c. 

Bestow  some  praise  on  Hamad  (e.g.,  compliment  house  and/or 
furnishings). 

d. 

Tell  Hamad  he  is  in  a  hurry  so  that  Hamad  does  not  feel  he  has 
to  waste  time  with  small  talk. 

3.  You  have  been  assigned  with  detennining  who  is  stealing  money  from  the  marketplace,  and 
are  talking  with  a  local  to  obtain  information.  However,  the  local  insists  you  purchase  some 
of  his  goods  before  he  can  help  you.  You  explain  you  do  not  have  the  money  but  that  you 
would  greatly  appreciate  his  help.  The  local  does  not  budge  and  continues  to  repeat  that  you 
should  buy  his  goods.  You  still  need  the  infonnation,  however.  What  do  you  do? 
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a. 

Stay  calm  and  be  patient.  Offer  him  help  or  other  services  for  his 
cooperation. 

b. 

Walk  away  and  find  someone  that  will  be  willing  to  help. 

c. 

Be  patient  and  wait.  This  will  show  him  that  you  are  serious. 

d. 

Change  the  topic  of  conversation  in  an  attempt  to  distract  him  from 
selling  his  goods. 

4.  Lieutenant  Colonel  Converse  is  meeting  Achmed  for  the  first  time,  and  they  are  getting  to 
know  each  other.  Converse  knows  that  Achmed  is  an  important  businessman  with  influence 
in  this  area.  To  develop  rapport  with  Achmed,  what  sorts  of  topics  of  conversation  would  be 
good  for  Lieutenant  Colonel  Converse  to  bring  up  during  the  meeting? 


a. 

How  much  authority  he  has  as  Lieutenant  Colonel  in  the  Army. 

b. 

Ask  Achmed  for  more  details  about  what  he  does  in  his  line  of 
work. 

c. 

Discuss  the  current  issues  with  the  construction  of  the  town  bridge. 

d. 

How  much  he  enjoys  the  local  food  in  this  area. 

5.  Major  O’Rourke  is  about  to  meet  with  a  local  leader  for  the  first  time.  The  Major  is 
concerned  about  the  potential  outcome  of  this  meeting  where  he  will  try  to  find  out 
information  about  suspected  security  issues  in  the  local  market  place.  What  should  you  tell 
him? 


a. 

Remind  him  that  the  leader  has  different  views  from  him.  It  helps  if 
he  acknowledges  the  leaders  perspective  before  introducing  his  own 
thoughts  on  the  matter. 

b. 

It  will  help  if  he  has  planned  for  the  possible  effects  of  both  success 
and  failure  of  the  meeting  on  the  area  of  operations. 

c. 

He  should  be  ready  to  put  pressure  on  the  local  leader  if  he  does  not 
immediately  provide  the  information  needed.  That  is  how  business  is 
done  in  this  town. 

d. 

He  should  focus  on  his  own  goals,  and  figure  out  a  way  to  get  the 
local  leader  to  bend  to  his  will.  This  will  speed  up  the  process. 

6.  Major  Collins  is  meeting  the  local  market  sellers  for  the  first  time.  They  have  been  arguing 
about  the  location  of  their  market  stalls.  The  Major  is  given  the  assignment  to  end  the 
feuding  so  the  market  can  reopen.  What  sorts  of  negotiation  techniques  can  he  use  to  end  the 
feud? 
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a. 

Tell  the  market  sellers  that  if  they  do  not  cooperate  they  will  not  be 
allowed  to  sell  their  products  at  the  market. 

b. 

Determine  the  different  interests  of  all  parties  and  attempt  to  align 
everyone’s  goals.  The  goal  is  to  have  a  situation  where  both  parties 
win  and  relationships  are  stronger. 

c. 

Determine  the  different  interests  of  the  townspeople  to  decide  the 
quickest  solution.  It  is  okay  if  there  is  a  losing  party  because  the  goal 
is  to  quickly  reopen  the  market. 

d. 

Observe  and  guide  the  townspeople  but  remain  removed  from  the 
situation. 

7.  The  local  schools  are  lacking  supplies  for  the  students.  You  find  out  that  the  neighboring 
town  has  storage  with  extra  school  supplies.  You  are  given  the  task  of  meeting  with  the 
leader  of  the  neighboring  town  and  getting  him  to  donate  some  school  supplies.  You  are  told 
he  is  not  a  generous  man.  What  can  you  do  to  get  his  cooperation? 


a. 

Find  out  what  his  needs  are  and  attempt  to  negotiate  a  solution  in 
exchange  for  the  school  supplies. 

b. 

Remind  him  that  by  not  helping  the  relationship  between  the  two  towns 
may  greatly  deteriorate. 

c. 

Get  to  know  him  and  build  a  relationship  before  requesting  the  school 
supplies. 

d. 

Offer  him  additional  security  for  his  community  in  exchange  for  the 
school  supplies. 

8.  Lieutenant  Rivera  has  set  up  a  meeting  with  the  local  Doctor  to  detennine  how  to  make  the 
clinic  more  efficient.  This  is  Lieutenant  Rivera’s  first  time  in  this  town  and  meeting  the 
Doctor.  What  is  the  best  advice  you  can  give  her? 


a. 

Don’t  waste  time  with  small  talk.  The  Doctor  has  many  patients. 

b. 

Let  him  lead  the  conversation.  He  will  tell  you  when  he  is  ready  to  talk 
business. 

c. 

Always  take  full  control  of  the  conversation.  Interrupt  him  if  you  feel 
he  is  moving  away  from  the  purpose  of  the  conversation. 

d. 

Begin  talking  about  yourself  and  afterwards  start  talking  business. 
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Research  Survey  1  (Feedback  Orientation  Scale) 


Please  answer  each  statement  by  indicating  the  degree  to  which  you  agree  or  disagree  from 
1  (strongly  disagree)  to  6  (strongly  agree)  by  circling  only  one  answer.  There  are  no  right  or  wrong 
answers.  Please  describe  yourself  honestly.  These  results  will  in  no  way  be  used  at  a  personal  or 
individual  level.  Your  anonymity  is  guaranteed. 


1  2 

Strongly  Moderately 

Disagree  Disagree 


3 

Somewhat 

Disagree 


4  5  6 

Somewhat  Moderately  Strongly 

Agree  Agree  Agree 


The  following  statements  concern  your  views  towards  feedback. 


1 .  Feedback  contributes  to  my  success  at  work. 

1 

2 

3 

4 

5 

6 

2.  To  develop  my  skills  at  work,  I  rely  on  feedback. 

1 

2 

3 

4 

5 

6 

3.  Feedback  is  critical  for  improving  performance. 

1 

2 

3 

4 

5 

6 

4.  Feedback  from  supervisors  can  help  me  advance  in  a  company. 

1 

2 

3 

4 

5 

6 

5.  I  find  that  feedback  is  critical  for  reaching  my  goals. 

1 

2 

3 

4 

5 

6 

1 .  It  is  my  responsibility  to  apply  feedback  to  improve  my 
performance. 

1 

2 

3 

4 

5 

6 

2.  I  hold  myself  accountable  to  respond  to  feedback  appropriately. 

1 

2 

3 

4 

5 

6 

3.  I  don’t  feel  a  sense  of  closure  until  I  respond  to  feedback. 

1 

2 

3 

4 

5 

6 

4.  If  my  supervisor  gives  me  feedback,  it  is  my  responsibility  to 
respond  to  it. 

1 

2 

3 

4 

5 

6 

5.  I  feel  obligated  to  make  changes  based  on  feedback. 

1 

2 

3 

4 

5 

6 

1. 

I  try  to  be  aware  of  what  other  people  think  of  me. 

1  2 

3 

4 

5 

6 

2. 

Using  feedback,  I  am  more  aware  of  what  people  think  of  me. 

1  2 

3 

4 

5 

6 

3. 

Feedback  helps  me  manage  the  impression  I  make  on  others. 

1  2 

3 

4 

5 

6 

4. 

Feedback  lets  me  know  how  I  am  perceived  by  others. 

1  2 

3 

4 

5 

6 

5. 

I  rely  on  feedback  to  help  me  make  a  good  impression. 

1  2 

3 

4 

5 

6 
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1  2 

Strongly  Moderately 

Disagree  Disagree 


3 

Somewhat 

Disagree 


4 

Somewhat 

Agree 


5  6 

Moderately  Strongly 
Agree  Agree 


The  following  statements  concern  your  views  towards  feedback. 


1. 

I  feel  self-assured  when  dealing  with  feedback. 

1  2 

3 

4 

5 

6 

2. 

Compared  to  others,  I  am  more  competent  at  handling  feedback. 

1  2 

3 

4 

5 

6 

3. 

I  believe  that  I  have  the  ability  to  deal  with  feedback  effectively. 

1  2 

3 

4 

5 

6 

4. 

I  feel  confident  when  responding  to  both  positive  and  negative 
feedback. 

1  2 

3 

4 

5 

6 

5. 

I  know  that  I  can  handle  the  feedback  that  I  receive. 

1  2 

3 

4 

5 

6 
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Research  Survey  2  (Manipulation  Check) 


Please  respond  to  each  statement  by  indicating  the  degree  to  which  you  agree  or  disagree  from  1 
(strongly  disagree)  to  6  (strongly  agree)  by  circling  only  one  answer. 


1 

Strongly 

Disagree 


2  3 

Moderately  Somewhat 

Disagree  Disagree 


4  5  6 

Somewhat  Moderately  Strongly 

Agree  Agree  Agree 


1. 

After  the  mission  was  over,  my  After  Action  Review  said  I  did  a 
good  job  on  the  mission. 

1 

2 

3 

4 

5 

6 

2. 

The  information  given  to  me  about  my  performance  showed  that  I 
interacted  poorly  with  the  characters. 

1 

2 

3 

4 

5 

6 

3. 

The  information  given  to  me  about  my  performance  showed  that  I 
did  well  interacting  with  the  characters. 

1 

2 

3 

4 

5 

6 

4. 

In  general,  the  voice  over  information  said  I  did  not  interact  with  the 
characters  very  well. 

1 

2 

3 

4 

5 

6 

5. 

The  information  I  received  about  my  performance  from  the  After 
Action  Review  screen  was  negative. 

1 

2 

3 

4 

5 

6 

6. 

The  information  about  my  performance  provided  by  the  voice  over 
was  generally  favorable. 

1 

2 

3 

4 

5 

6 

1. 

Generally,  after  my  conversation  with  each  character  the 
coach/voiceover  gave  me  information  about  my  performance. 

1 

2 

3 

4 

5 

6 

2. 

I  received  information  about  my  performance  several  times  during  the 
mission  (as  a  voiceover)  and  then  at  the  end  (After  Action  Review 
screen). 

1 

2 

3 

4 

5 

6 

3. 

I  knew  how  I  was  performing  throughout  the  game  and  not  just  at  the 
end  of  the  mission. 

1 

2 

3 

4 

5 

6 

4. 

I  was  given  information  about  my  performance  a  couple  of  times 
during  the  game. 

1 

2 

3 

4 

5 

6 

5. 

The  coach/voiceover  rarely  provided  me  with  information  about  my 
performance  while  I  was  playing  the  videogame. 

1 

2 

3 

4 

5 

6 
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6. 

I  did  not  receive  any  information  about  my  performance  until  the 

After  Action  Review  screen  at  the  end  of  the  mission. 

1 

2 

3 

4 

5 

6 

7. 

I  generally  did  not  know  how  well  I  did  on  the  game  until  the  end  of 
the  mission. 

1 

2 

3 

4 

5 

6 

8. 

I  was  not  provided  with  information  about  my  performance  during  the 
game. 

1 

2 

3 

4 

5 

6 
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Research  Survey  3  (Attention  Scale) 


Please  respond  to  each  statement  by  indicating  the  degree  to  which  you  agree  or  disagree  from 
1  (strongly  disagree)  to  8  (strongly  agree)  by  circling  only  one  answer. 


1 

1 

1 

2 

1 

3 

1 

4 

1 

5 

6 

7 

1 

8 

Never 

Constantly 

1. 

I  paid  close  attention  to  the  kind  of  errors  I  was  making. 

1  2 

3 

4 

5 

6 

7 

8 

2. 

I  focused  my  attention  on  whatever  was  going  wrong. 

1  2 

3 

4 

5 

6 

7 

8 

3. 

I  focused  my  total  attention  on  learning  a  specific  rule. 

1  2 

3 

4 

5 

6 

7 

8 

4.  I  focused  my  attention  on  being  ready  for  a  change  in  the 
conversation. 

1  2 

3 

4 

5 

6 

7 

8 

5.  I  focused  my  attention  on  responding  to  the  behavior  changes  of  the 
characters. 

1  2 

3 

4 

5 

6 

7 

8 

6. 

I  thought  about  new  strategies  for  improving  my  performance. 

1  2 

3 

4 

5 

6 

7 

8 

7. 

I  thought  ahead  to  what  I  would  do  next  to  improve  my  performance. 

1  2 

3 

4 

5 

6 

7 

8 

8. 

I  told  myself  things  to  encourage  me  to  try  harder. 

1  2 

3 

4 

5 

6 

7 

8 

9. 

I  focused  my  total  attention  on  how  fast  I  could  solve  the  mission. 

1  2 

3 

4 

5 

6 

7 

8 

10. 1  focused  my  total  attention  on  passing  as  many  objectives  as 
possible. 

1  2 

3 

4 

5 

6 

7 

8 

1. 

I  took  “mental  breaks”  during  the  task. 

1 

2 

3 

4 

5 

6 

7 

8 

2. 

I  daydreamed  while  doing  the  task. 

1 

2 

3 

4 

5 

6 

7 

8 

3. 

I  lost  interest  in  the  task  for  short  periods. 

1 

2 

3 

4 

5 

6 

7 

8 

4. 

I  thought  about  other  things  that  I  have  to  do. 

1 

2 

3 

4 

5 

6 

7 

8 

5. 

I  wondered  about  how  my  performance  compared  with  others. 

1 

2 

3 

4 

5 

6 

7 

8 

6. 

I  thought  about  the  difficulty  of  the  task. 

1 

2 

3 

4 

5 

6 

7 

8 
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Presence  Questionnaire 


For  the  following  questions,  please  circle  the  number  which  best  represents  your  experience. 

1.  Please  rate  your  sense  of  being  in  the  computer-generated  world,  where  7  represents  your 
nonnal  experience  of  being  in  a  place. 

I  had  a  sense  of  '‘being  there  ’’  in  the  computer-generated  world: 

1  2  3  4  5  6  7 

Not  at  all  Very 

Much 


2.  To  what  extent  were  there  times  during  the  experience  when  the  computer-generated 
world  was  the  reality  for  you? 

There  were  times  during  the  experience  when  the  virtual  environment  was  the  reality  for 
me: 


1  2  3  4  5 

At  no  time 


Almost  all 
the  time 


3.  When  you  think  back  to  the  experience,  do  you  think  of  the  computer-generated  world 
more  as  images  that  you  saw  or  as  somewhere  that  you  visited? 

The  computer-generated  world  seems  to  me  to  be  more  like: 


12  3  4 

Images 
that  I  saw 


5  6  7 

Somewhere 
that  I 
visited 


4.  During  the  time  of  the  experience,  which  was  the  strongest  on  the  whole,  your  sense  of 
being  in  the  computer-generated  world  or  of  being  elsewhere? 

I  had  a  stronger  sense  of: 


1  2  3  4  5 

Being 
elsewhere 


6  7 

Being  in  the 
virtual 
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environment 


General  Information 


Gender:  Male  _  Female  _  Have  you  ever  participated  in  a  computer-based 

interpersonal  skills  training  before? 

Age:  _  A.  yes 

B.  no 

How  would  you  describe  yourself?  If  yes,  please  describe  it: 

A.  Asian/Pacific  Islander 

B.  African  American 

C.  Hispanic/Latino 

D.  Caucasian 

E.  Other  _ 

Have  you  previously  participated  in  a 
videogame  based  negotiations  training 
(i.e.  Elect  Bilat)  at  the  University  of 
Central  Florida? 

A.  yes 

B.  no  How  often  do  you  play  computer-based 

videogames? 

A.  Never 

B.  A  few  times  a  year 

C.  2-3  times  a  month 

D.  2-3  times  a  week 

E.  Every  day 
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Appraisal  Inventory  (Self-efficacy  scale) 


Think  about  a  situation  where  you  may  need  to  communicate  effectively  with  another  person. 
Rate  your  confidence  in  successfully  using  each  tactic  listed  below  when  communicating  with 
someone. 

Rate  your  degree  of  confidence  by  recording  a  number  from  0  to  100  using  the  scale  given 
below: 

0  10  20  30  40  50  60  70  80  90  100 

Cannot  Moderately  Highly  certain 

do  at  all  can  do  can  do 


Confidence 

(0  - 100) 

Establish  a  high  level  of  rapport  with  the  other  person  _ 

Convince  the  other  person  to  agree  with  me  _ 

Find  tradeoffs  that  will  benefit  both  me  and  the  other  person  _ 

Look  for  an  agreement  that  maximizes  both  of  our  interests’  _ 

Use  appropriate  greetings  when  meeting  the  other  person  _ 

Build  a  relationship  by  engaging  in  small  talk  _ 

Take  control  of  the  conversation  by  only  discussing  the  topics  I  choose  _ 

Acknowledge  the  other  persons  perspective  during  a  disagreement  _ 

Remain  patient  during  a  heated  disagreement  _ 

Reduce  tension  and  conflict  during  a  disagreement  _ 
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